Page 54 of 61

Re: Spiders & bots to add to phpBB

Posted: Fri Mar 01, 2013 3:15 pm
by J_M
why do you think this requires you to even know about it, much less have to do something about it?
okay, now you are just picking on me.... just kidd'n : )

My site runs on a cloud server and over the past couple months it has run into problems with resource overload. Having a botnet such as 360spider that sends multiple computers is a good example. Fortunately, my host was able to make some allocation changes and it's running well now. But as a result of the problems I did have to spend a bunch of time dealing with the problem.

I am using a QA that is working for bots, but humans are another issue. My test blocking is a broader geographic range than I would like but may help with some of these problems. Because humans will always be able to get through, I now use QA plus Admin allow for the registration, this uses up time.

Why also would I freely allow email scrapers to run all over the forum if I don't need too? Protect the users.

I'm sure there are other issues that I am leaving out.
why do you think this requires you to even know about it
Why not?

regards,

Re: Manage_Bots 6.0 Beta Test

Posted: Fri Mar 01, 2013 9:33 pm
by Muad''Dib
Pony99CA wrote:I have created manage_bots 6.0, a major upgrade, and would like some beta testing. Please do not download the attached file unless you want to help test this.

Besides additions and updates to the bot list, there are several major changes and updates.

The major updates are:
  • The level system has been completely redone, which means that the level numbers have all changed. They are now based on the 'class" of bot (phpBB standard, major search engine, minor search engine, Web tools, etc.) instead of the bot's reporter.
  • A new flags option has been added to allow greater control over which bots are changed. Boolean logic is used instead of simple <= or >= testing. This is mutually exclusive with the level option.
  • A new reporter option has been added to allow the specified operation to only apply to bots reported by one person.
The full list of updates includes:
  • Added Flags option to control adding, deleting, activating and deactivating bots more precisely than using Level
  • Added Reporter option to control adding, deleting, activating and deactivating bots by the user who reported them
  • Changed Level parameter to be function-based; user-based bot actions can be done with the new Reporter option
  • Changed default Level parameter to 128 in conjunction with previous change
  • Added total counts to credits
  • Added information on problem reporting to help
  • Added Flipboard, Genieo, Semiocast & Twitmunin bots from Marcus Wendel
  • Added InboundScore, MySmutSearch, OSS, Solomono, Wotbox & Yahoo DoCoMo bots from _Vinny_
  • Added Aboundex, Bing Preview, Botje, CheckParams, Download Ninja, Panopta, Search Web Engine, SiteIntel, SitesLikeIt, Supybot, URLDBCleaner & WASALive bots from AmigoJack
  • Added Grapeshot bot from roBBx
  • Added CloudACL bot from Schwpz
  • Updated URL for Xaldon
  • Updated URL for NerdByNature (thanks, Marcus)
  • Updated formatting of Help command display to use standard HTML headers, added Command Format and Parameter headers and used dictionary lists for parameters
  • Simplified command line parameter processing with new check_option function
  • Moved Level parameter checking into command-specific option checking areas
  • Updated bot cache handling to do it once per command (it could be cleared twice if bots were updated)
  • Updated bot cache handling to reload cache, not just clear it
  • Replaced array index variables with true constants (thanks, AmigoJack)
  • Replaced script version number, visiting/non-visiting level number and other variables with true constants
  • Created common match_bot bot matching function
  • Fixed bots array level/credit mapping
  • Fixed bug in list_bots where reporter output had an incorrect single quote in middle of string
  • Fixed bug where using request_var in get_parameter caused errors in parameters to be missed; new check_option works around that
  • Updated debug statements to use "magic" constants (like __FUNCTION__)
  • Fixed debug formatting in list_bots
  • Fixed debug output in List Format and Number parameter processing
  • Removed unnecessary references to $config phpBB global variable
  • Removed unnecessary TRUE argument calling delete_bots
For help after uploading the script, type something like http://example.com/phpbb3/manage_bots.php?? (note the two question marks!).

I have tested this, but want additional testing before I consider the release final. Please do not post questions or bug reports here; PM me if you find any bugs or have any questions. If enough questions about something come up, I will post them here.

Steve

Just tried this today and it worked fine.. It updated a few bots and added a few. It did however show the following at the bottom of the output:

Code: Select all

[phpBB Debug] PHP Notice: in file [ROOT]/manage_bots.php on line 1817: Undefined index: MBOTS_ADDED_LOG
I ran it again after that and it didn't give me the error.

Re: Manage_Bots 6.0 Beta Test

Posted: Sat Mar 02, 2013 2:42 am
by Pony99CA
Muad''Dib wrote:Just tried this today and it worked fine.. It updated a few bots and added a few. It did however show the following at the bottom of the output:

Code: Select all

[phpBB Debug] PHP Notice: in file [ROOT]/manage_bots.php on line 1817: Undefined index: MBOTS_ADDED_LOG
I ran it again after that and it didn't give me the error.
First, thanks for trying manage_bots 6.0 out. I haven't gotten nearly enough feedback on it. :)

However, did you see this:
Please do not post questions or bug reports here; PM me if you find any bugs or have any questions.
Anyway, just in case I've updated the code since that version, please PM me the code around line 1817 (+/- 10 lines or so or to the beginning/end of the current function, whichever is less). Also let me know the exact command that you ran.

Not getting the error when running it again makes sense as there wouldn't be any more bots to add (unless you didn't add them all the first time and tried to add more the second time).

Thanks.

Steve

P.S. You didn't really need to quote my entire post (or any of it). Just saying "I tried testing manage_bots 6.0 and...." would have been more than enough. ;) (I know, that Arrakis sun probably messes with the brain. :lol:)

Re: Spiders & bots to add to phpBB

Posted: Thu Mar 07, 2013 7:28 pm
by Marcus Wendel
Bot name: Tweeted Times [Bot]
Agent match: TweetedTimes
User agent string: Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)
Website: http://www.tweetedtimes.com

Bot name: Bing Preview [Bot]
Agent match: BingPreview
User agent string: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Website: http://www.bing.com
(This one existed without the agent string I think)

Bot name: MetaURI [Bot]
Agent match: MetaURI
User agent string: MetaURI API/2.0 +metauri.comBingPreview/1.0b
Website: http://www.metauri.com
(This one existed without the agent string I think)

/Marcus

Re: Spiders & bots to add to phpBB

Posted: Wed Mar 13, 2013 8:56 pm
by roBBx
Marcus Wendel wrote:Bot name: 360sou [Bot]
Agent match: 360Spider
User agent string: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Website: http://www.360sou.com
Just seen it with User agent string:

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider

Perhaps it's better to use only "360Spider" string to identify it.

Re: Spiders & bots to add to phpBB

Posted: Thu Mar 14, 2013 1:35 am
by Pony99CA
roBBx wrote:
Marcus Wendel wrote:Bot name: 360sou [Bot]
Agent match: 360Spider
User agent string: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Website: http://www.360sou.com
Just seen it with User agent string:

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider

Perhaps it's better to use only "360Spider" string to identify it.
He did -- that's what the Agent match is for. Seeing the entire User Agent can sometimes give additional interesting information, like Web links or E-mail addresses, so I appreciate it when people post that, too.

Steve

Re: Spiders & bots to add to phpBB

Posted: Fri Mar 15, 2013 11:40 pm
by roBBx
Oh yes, it was already correct, sorry for mistake! This is a very invasive bot, it opened a lot of sessions with different IPs before I defined the new bot. :roll:

Re: Spiders & bots to add to phpBB

Posted: Tue Mar 19, 2013 7:44 pm
by @Marcin
Bot name: Google Developers
Agent match: +https://developers.google.com/+/web/snippet

Re: Spiders & bots to add to phpBB

Posted: Sun Mar 24, 2013 6:16 am
by Marcus Wendel
Bot name: Akregator [Bot]
Agent match: Akregator
User agent string: Akregator/4.10.1; syndication
Website: http://akregator.sourceforge.net/index.php

Bot name: Grapeshot [Bot]
Agent match: Grapeshot
User agent string: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Website: http://www.grapeshot.co.uk/crawler.php
(existed before without user agent string)

/Marcus

Re: Spiders & bots to add to phpBB

Posted: Thu Mar 28, 2013 12:48 pm
by MaFeSa
Tested manage_bots 6.0 and is perfect!
:D

Many thanks ;)

Re: Spiders & bots to add to phpBB

Posted: Thu Mar 28, 2013 8:54 pm
by Marcus Wendel
Bot name: Google+ Snippet [Bot]
Agent match: developers.google.com/+/web/snippet/
User agent string: Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Website: https://developers.google.com/+/web/snippet/

/Marcus

Re: Spiders & bots to add to phpBB

Posted: Fri Mar 29, 2013 6:35 pm
by Marcus Wendel
This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
/Marcus

Re: Spiders & bots to add to phpBB

Posted: Fri Mar 29, 2013 8:02 pm
by Pony99CA
Marcus Wendel wrote:This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
Yeah, that's not very useful.

Did you check the IP address?

Steve

Re: Spiders & bots to add to phpBB

Posted: Fri Mar 29, 2013 8:09 pm
by Marcus Wendel
Pony99CA wrote:
Marcus Wendel wrote:This bot was not so helpful:
Guest IP: 173.231.146.52 » Whois
Crawler
Yeah, that's not very useful.

Did you check the IP address?

Steve
Nothing helpful, it apparently belongs to www.voxel.net.

/Marcus

Re: Spiders & bots to add to phpBB

Posted: Sat Apr 06, 2013 2:53 am
by techman41973
I appreciate all of the work that went into these lists and script.
But can anyone explain the advantages of adding this long list of new bots to my phpbb site?
Have all or most of these bots been tested to be beneficial in some way?
I'm not sure how a bot (if perhaps it was a spam bot) could negatively effect my site.