Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
J_M
Registered User
Posts: 258
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M » Fri Mar 01, 2013 3:15 pm

why do you think this requires you to even know about it, much less have to do something about it?
okay, now you are just picking on me.... just kidd'n : )

My site runs on a cloud server and over the past couple months it has run into problems with resource overload. Having a botnet such as 360spider that sends multiple computers is a good example. Fortunately, my host was able to make some allocation changes and it's running well now. But as a result of the problems I did have to spend a bunch of time dealing with the problem.

I am using a QA that is working for bots, but humans are another issue. My test blocking is a broader geographic range than I would like but may help with some of these problems. Because humans will always be able to get through, I now use QA plus Admin allow for the registration, this uses up time.

Why also would I freely allow email scrapers to run all over the forum if I don't need too? Protect the users.

I'm sure there are other issues that I am leaving out.
why do you think this requires you to even know about it
Why not?

regards,

User avatar
Muad''Dib
Registered User
Posts: 311
Joined: Tue Jun 12, 2007 6:20 pm
Contact:

Re: Manage_Bots 6.0 Beta Test

Post by Muad''Dib » Fri Mar 01, 2013 9:33 pm

Pony99CA wrote:I have created manage_bots 6.0, a major upgrade, and would like some beta testing. Please do not download the attached file unless you want to help test this.

Besides additions and updates to the bot list, there are several major changes and updates.

The major updates are:
  • The level system has been completely redone, which means that the level numbers have all changed. They are now based on the 'class" of bot (phpBB standard, major search engine, minor search engine, Web tools, etc.) instead of the bot's reporter.
  • A new flags option has been added to allow greater control over which bots are changed. Boolean logic is used instead of simple <= or >= testing. This is mutually exclusive with the level option.
  • A new reporter option has been added to allow the specified operation to only apply to bots reported by one person.
The full list of updates includes:
  • Added Flags option to control adding, deleting, activating and deactivating bots more precisely than using Level
  • Added Reporter option to control adding, deleting, activating and deactivating bots by the user who reported them
  • Changed Level parameter to be function-based; user-based bot actions can be done with the new Reporter option
  • Changed default Level parameter to 128 in conjunction with previous change
  • Added total counts to credits
  • Added information on problem reporting to help
  • Added Flipboard, Genieo, Semiocast & Twitmunin bots from Marcus Wendel
  • Added InboundScore, MySmutSearch, OSS, Solomono, Wotbox & Yahoo DoCoMo bots from _Vinny_
  • Added Aboundex, Bing Preview, Botje, CheckParams, Download Ninja, Panopta, Search Web Engine, SiteIntel, SitesLikeIt, Supybot, URLDBCleaner & WASALive bots from AmigoJack
  • Added Grapeshot bot from roBBx
  • Added CloudACL bot from Schwpz
  • Updated URL for Xaldon
  • Updated URL for NerdByNature (thanks, Marcus)
  • Updated formatting of Help command display to use standard HTML headers, added Command Format and Parameter headers and used dictionary lists for parameters
  • Simplified command line parameter processing with new check_option function
  • Moved Level parameter checking into command-specific option checking areas
  • Updated bot cache handling to do it once per command (it could be cleared twice if bots were updated)
  • Updated bot cache handling to reload cache, not just clear it
  • Replaced array index variables with true constants (thanks, AmigoJack)
  • Replaced script version number, visiting/non-visiting level number and other variables with true constants
  • Created common match_bot bot matching function
  • Fixed bots array level/credit mapping
  • Fixed bug in list_bots where reporter output had an incorrect single quote in middle of string
  • Fixed bug where using request_var in get_parameter caused errors in parameters to be missed; new check_option works around that
  • Updated debug statements to use "magic" constants (like __FUNCTION__)
  • Fixed debug formatting in list_bots
  • Fixed debug output in List Format and Number parameter processing
  • Removed unnecessary references to $config phpBB global variable
  • Removed unnecessary TRUE argument calling delete_bots
For help after uploading the script, type something like http://example.com/phpbb3/manage_bots.php?? (note the two question marks!).

I have tested this, but want additional testing before I consider the release final. Please do not post questions or bug reports here; PM me if you find any bugs or have any questions. If enough questions about something come up, I will post them here.

Steve

Just tried this today and it worked fine.. It updated a few bots and added a few. It did however show the following at the bottom of the output:

Code: Select all

[phpBB Debug] PHP Notice: in file [ROOT]/manage_bots.php on line 1817: Undefined index: MBOTS_ADDED_LOG
I ran it again after that and it didn't give me the error.

Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Manage_Bots 6.0 Beta Test

Post by Pony99CA » Sat Mar 02, 2013 2:42 am

Muad''Dib wrote:Just tried this today and it worked fine.. It updated a few bots and added a few. It did however show the following at the bottom of the output:

Code: Select all

[phpBB Debug] PHP Notice: in file [ROOT]/manage_bots.php on line 1817: Undefined index: MBOTS_ADDED_LOG
I ran it again after that and it didn't give me the error.
First, thanks for trying manage_bots 6.0 out. I haven't gotten nearly enough feedback on it. :)

However, did you see this:
Please do not post questions or bug reports here; PM me if you find any bugs or have any questions.
Anyway, just in case I've updated the code since that version, please PM me the code around line 1817 (+/- 10 lines or so or to the beginning/end of the current function, whichever is less). Also let me know the exact command that you ran.

Not getting the error when running it again makes sense as there wouldn't be any more bots to add (unless you didn't add them all the first time and tried to add more the second time).

Thanks.

Steve

P.S. You didn't really need to quote my entire post (or any of it). Just saying "I tried testing manage_bots 6.0 and...." would have been more than enough. ;) (I know, that Arrakis sun probably messes with the brain. :lol:)
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Thu Mar 07, 2013 7:28 pm

Bot name: Tweeted Times [Bot]
Agent match: TweetedTimes
User agent string: Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)
Website: http://www.tweetedtimes.com

Bot name: Bing Preview [Bot]
Agent match: BingPreview
User agent string: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Website: http://www.bing.com
(This one existed without the agent string I think)

Bot name: MetaURI [Bot]
Agent match: MetaURI
User agent string: MetaURI API/2.0 +metauri.comBingPreview/1.0b
Website: http://www.metauri.com
(This one existed without the agent string I think)

/Marcus

roBBx
Registered User
Posts: 287
Joined: Fri Feb 15, 2008 3:00 am
Contact:

Re: Spiders & bots to add to phpBB

Post by roBBx » Wed Mar 13, 2013 8:56 pm

Marcus Wendel wrote:Bot name: 360sou [Bot]
Agent match: 360Spider
User agent string: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Website: http://www.360sou.com
Just seen it with User agent string:

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider

Perhaps it's better to use only "360Spider" string to identify it.

Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA » Thu Mar 14, 2013 1:35 am

roBBx wrote:
Marcus Wendel wrote:Bot name: 360sou [Bot]
Agent match: 360Spider
User agent string: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Website: http://www.360sou.com
Just seen it with User agent string:

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider

Perhaps it's better to use only "360Spider" string to identify it.
He did -- that's what the Agent match is for. Seeing the entire User Agent can sometimes give additional interesting information, like Web links or E-mail addresses, so I appreciate it when people post that, too.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

roBBx
Registered User
Posts: 287
Joined: Fri Feb 15, 2008 3:00 am
Contact:

Re: Spiders & bots to add to phpBB

Post by roBBx » Fri Mar 15, 2013 11:40 pm

Oh yes, it was already correct, sorry for mistake! This is a very invasive bot, it opened a lot of sessions with different IPs before I defined the new bot. :roll:

User avatar
@Marcin
Registered User
Posts: 19
Joined: Wed Feb 01, 2012 12:05 pm
Location: Milton Keynes
Name: Marcin
Contact:

Re: Spiders & bots to add to phpBB

Post by @Marcin » Tue Mar 19, 2013 7:44 pm

Bot name: Google Developers
Agent match: +https://developers.google.com/+/web/snippet

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Sun Mar 24, 2013 6:16 am

Bot name: Akregator [Bot]
Agent match: Akregator
User agent string: Akregator/4.10.1; syndication
Website: http://akregator.sourceforge.net/index.php

Bot name: Grapeshot [Bot]
Agent match: Grapeshot
User agent string: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Website: http://www.grapeshot.co.uk/crawler.php
(existed before without user agent string)

/Marcus

MaFeSa
Registered User
Posts: 174
Joined: Wed Feb 11, 2009 7:48 am

Re: Spiders & bots to add to phpBB

Post by MaFeSa » Thu Mar 28, 2013 12:48 pm

Tested manage_bots 6.0 and is perfect!
:D

Many thanks ;)
Last edited by MaFeSa on Fri Mar 29, 2013 9:18 am, edited 1 time in total.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Thu Mar 28, 2013 8:54 pm

Bot name: Google+ Snippet [Bot]
Agent match: developers.google.com/+/web/snippet/
User agent string: Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Website: https://developers.google.com/+/web/snippet/

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Fri Mar 29, 2013 6:35 pm

This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
/Marcus

Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA » Fri Mar 29, 2013 8:02 pm

Marcus Wendel wrote:This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
Yeah, that's not very useful.

Did you check the IP address?

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Fri Mar 29, 2013 8:09 pm

Pony99CA wrote:
Marcus Wendel wrote:This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
Yeah, that's not very useful.

Did you check the IP address?

Steve
Nothing helpful, it apparently belongs to www.voxel.net.

/Marcus

techman41973
Registered User
Posts: 410
Joined: Thu Mar 28, 2013 10:27 pm

Re: Spiders & bots to add to phpBB

Post by techman41973 » Sat Apr 06, 2013 2:53 am

I appreciate all of the work that went into these lists and script.
But can anyone explain the advantages of adding this long list of new bots to my phpbb site?
Have all or most of these bots been tested to be beneficial in some way?
I'm not sure how a bot (if perhaps it was a spam bot) could negatively effect my site.

Post Reply

Return to “phpBB Discussion”