Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: Spiders & bots to add to phpbb3

Post by ToonArmy » Wed Feb 17, 2010 10:57 pm

MPG187 wrote:
Dog Cow wrote:
MPG187 wrote:http://www.ip2location.com/65.55.109.155

I have had a visitor from that IP. The ISP is Microsoft, but it's from NY (MS is in Redmond). Is that a bot or does MS run an ISP?
Do you suppose that the database is incorrect? That's much more likely.

Why don't you look up Microsoft's netblock? See what that gives. Much more accurate results, no doubt!
If the site's not correct, then can you recommend a better one? I have been using it for a while and most of the time it gives good results...
The IP appears to be in a NYC data centre, just because Microsoft HQ is in Redmond that doesn't mean everything they do is. Don't put all your eggs in one basket and all that. ;)
Chris SmithBlogGitHub
Image
In a foreign field he lay. Lonely soldier unknown grave. On his dying words he prays. Tell the world of Paschendale.

User avatar
MPG187
Registered User
Posts: 519
Joined: Sun Apr 12, 2009 7:39 pm
Location: Operation420.net
Name: Operation 420
Contact:

Re: Spiders & bots to add to phpbb3

Post by MPG187 » Wed Feb 17, 2010 11:12 pm

So what would an NYC data center computer be doing browsing my site?
phpBB 3.0.4, http://operation420.net/forum, download package from phpBB.com (fresh install), mods installed, British English, MySQL 4.1.14

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpbb3

Post by Pony99CA » Thu Feb 18, 2010 2:21 am

clava wrote:What I mean in the message above is that after I ran add_bots.php there was no bot displayed on the section "Who is online". Only registered users are displayed there.
Unless you've changed something, bots are considered registered and displayed in the WHO IS ONLINE section. Look at the phpBB.com page and you'll likely see a few bots listed there.
clava wrote:But I must say there are a lot of bots visiting my forum, and I see those bots as visitors (logged as administrator I can see visitors agent and some of those visitors had msn, googlebot, yahoo! slurp, etc agents.)
Where are you seeing those agents? If you're clicking on WHO IS ONLINE as an admin, the agents show up there (as I mentioned previously).

Please use the exact names of pages and fields (using the default language pack) when reporting things. Making people guess what you mean just delays things.
clava wrote:This was yesterday. Today still no bots on "who is online".
Again, you'll only see them listed if they're currently online or were online within the time specified by the setting that I pointed out. If your board isn't very well-known, bots might not hang out there for long periods of time.
clava wrote:PS: I deleted now all bots and I added some of them manualy and it is working now.
Maybe I will add the entire list without '52', '1', '999' and I guess this way will work.
I'm not sure what "without '52', '1', '999'" means. There's only one thing that you need to do to change the level of bots added (either specify the &level= parameter on the call or change the $default_level variable).

If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpbb3

Post by Pony99CA » Thu Feb 18, 2010 2:24 am

clava wrote:
Pony99CA wrote:... If you find one, please post the details here (see previous posts for an idea of how they look).
And a bot that I haven't saw on your list but visited my forum:
agent: Mozilla/5.0+(compatible;+OkidokiBot/0.1;++http://www.okidoki.ro/bot.html)
Thanks. I'll add that to the list when I get time. (Are you in Romania, by any chance?)

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpbb3

Post by Pony99CA » Thu Feb 18, 2010 2:29 am

MPG187 wrote:So what would an NYC data center computer be doing browsing my site?
It could be an employee who works at some Micorosoft office in New York.

Please, let's not hijack this thread to guess why various IP addresses may show up on your site. Most bots will present an obvious agent string, and those are the ones that we want to capture.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
MPG187
Registered User
Posts: 519
Joined: Sun Apr 12, 2009 7:39 pm
Location: Operation420.net
Name: Operation 420
Contact:

Re: Spiders & bots to add to phpbb3

Post by MPG187 » Thu Feb 18, 2010 3:08 am

Pony99CA wrote:
MPG187 wrote:So what would an NYC data center computer be doing browsing my site?
It could be an employee who works at some Micorosoft office in New York.

Please, let's not hijack this thread to guess why various IP addresses may show up on your site. Most bots will present an obvious agent string, and those are the ones that we want to capture.

Steve
When I posted I thought they mighta been a bot. It guess MS Employees love my site. Or maybe they don't like the modded Xbox stuff. Anyways, I'll end it here...
phpBB 3.0.4, http://operation420.net/forum, download package from phpBB.com (fresh install), mods installed, British English, MySQL 4.1.14

User avatar
onehundredandtwo
Registered User
Posts: 1228
Joined: Fri Nov 14, 2008 8:07 am

Re: Spiders & bots to add to phpbb3

Post by onehundredandtwo » Thu Feb 18, 2010 6:28 am

I have had MSNBot view my website with an Internet Explorer user-agent before, and I doubt anyone from Microsoft would have found that website. My best guess is that Microsoft has some type of bot that checks if websites suspected of spamming / new websites are serving different content for bots.
Need help preventing spam? Read Preventing spam in phpBB 3.0.6 and above

User avatar
ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: Spiders & bots to add to phpbb3

Post by ToonArmy » Thu Feb 18, 2010 8:56 am

MPG187 wrote:So what would an NYC data center computer be doing browsing my site?
Indexing it? It's a MSN bot IP: 155.109.55.65.in-addr.arpa domain name pointer msnbot-65-55-109-155.search.msn.com.
Chris SmithBlogGitHub
Image
In a foreign field he lay. Lonely soldier unknown grave. On his dying words he prays. Tell the world of Paschendale.

User avatar
clava
Registered User
Posts: 23
Joined: Sun Oct 07, 2007 3:25 pm

Re: Spiders & bots to add to phpbb3

Post by clava » Thu Feb 18, 2010 2:35 pm

Pony99CA wrote: If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)

Steve
Hello,
I am very sory I could not make myself understood.
My english it is not so good.
Anyway, I have sent you a PM with detailed actions I made on my bot list and a link to my working forum with a lot of bots displayed on "Who is online".

User avatar
MPG187
Registered User
Posts: 519
Joined: Sun Apr 12, 2009 7:39 pm
Location: Operation420.net
Name: Operation 420
Contact:

Re: Spiders & bots to add to phpbb3

Post by MPG187 » Fri Feb 19, 2010 12:11 am

Saying that it's an MSN bot does make sense. But it doesn't show up as the MSN bot.

Unless is MSN using another IP address, one that is not registered with phpbb yet?

Next time the MSN bot it on my site I'll have to check it's user agent. I do remember when that visitor was on my site it was using IE...

EDIT (Not Double Posting)
Guest
IP: 75.101.184.216 » Whois
grvcrawler/0.3
phpBB 3.0.4, http://operation420.net/forum, download package from phpBB.com (fresh install), mods installed, British English, MySQL 4.1.14

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpbb3

Post by Pony99CA » Sun Feb 21, 2010 3:43 am

clava wrote:
Pony99CA wrote: If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)
Anyway, I have sent you a PM with detailed actions I made on my bot list and a link to my working forum with a lot of bots displayed on "Who is online".
After PM discussion with clava, he mentioned that he is using the following MODs (among others):
  • Override_the_edit_time_limit-1-0-1
  • top_posters_1_0_0a
  • users_online_time_102
  • guests_online_past_24_hours
  • activity_stats_mod
I don't know those MODs, but I suspect there could be some interaction with some of those. If anybody using those has run the add_bots script, have you noticed any problems?

Thanks,
Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpbb3

Post by Pony99CA » Sun Feb 21, 2010 10:39 am

MPG187 wrote:Saying that it's an MSN bot does make sense. But it doesn't show up as the MSN bot.

Unless is MSN using another IP address, one that is not registered with phpbb yet?

Next time the MSN bot it on my site I'll have to check it's user agent. I do remember when that visitor was on my site it was using IE...

EDIT (Not Double Posting)
Guest
IP: 75.101.184.216 » Whois
grvcrawler/0.3
That's not from MSN; that's the Amazon Electric Cloud bot that I reported on February 8. It's got a completely different IP address, but it resolves to the Electric Cloud nonetheless.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: Spiders & bots to add to phpbb3

Post by ToonArmy » Sun Feb 21, 2010 2:56 pm

Pony99CA wrote:That's not from MSN; that's the Amazon Electric Cloud bot that I reported on February 8. It's got a completely different IP address, but it resolves to the Electric Cloud nonetheless.
Those were two posts in one. ;)
Chris SmithBlogGitHub
Image
In a foreign field he lay. Lonely soldier unknown grave. On his dying words he prays. Tell the world of Paschendale.

maxwell2
Registered User
Posts: 150
Joined: Fri Apr 10, 2009 1:43 pm

Re: Spiders & bots to add to phpbb3

Post by maxwell2 » Sun Feb 21, 2010 7:55 pm

How do I add a bot to get a message that such a bot already exists :shock:
How to remove all the bots and add new ones? Modification is to add mass bots? :)

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

New Manage_Bots Script 5.5

Post by Pony99CA » Mon Feb 22, 2010 12:14 am

manage_bots has been updated to version 5.5. manage_bots is basically version 4.0 of add_bots with the functions of delete_bots and list_bots merged in. This makes things simpler for me as I don't have to keep the bots tables in sync and simpler for you because there's only one script to worry about.

To install and use manage_bots, please read the comments at the top of the script.

New features and changes over the original add_bots script include the following (new features are in bold):
  • Combined add_bots, delete_bots and list_bots into one manage_bots script
  • Adding bots now only shows the bots added unless you specify the ListAll option.
  • Added the update operation to just invoke the update_bots function
  • Added new command to check for conflicting bots
  • Added Help (or "?") command to display command help; the "?" works even when not logged in to help you get started
  • Added credits, version number and link to current script to help
  • Updated security to only allow board administrators with manage bots permission to run this script.
  • Added ability to set bot's active status and last visit timestamp. Combined with the List enhancements below, you can now use Delete without losing your Bot visit data.
  • Added ability to set bot's E-mail address and Web site (based on an idea by AmigoJack). This is only viewable in the ACP user view (unless you create a MOD to allow viewing Bot profiles).
  • Updated List command to dump bots' new fields listed above (doing a List before a Delete allows restoring that data after the delete).
  • Added optional formatting for List output and added format and number options
  • Added Activate and Deactivate commands to change bots activation status quickly.
  • Added password support to prevent unauthorized users from running the script
  • Added warning if conflicting agent strings or IP addresses are found
  • Moved non-debugging strings into array in customization section suitable for translation
  • Added command and parameter names to language array to allow translating them
  • Added checks for maximum values of some parameters
  • Cleared the bot cache so results take effect immediately (I hope)
  • Fixed bug in updating if new bot name already existed
  • Moved checks to see if the bot was in the user table to the bot_defined function
  • Used clean username for add and delete comparisons to avoid collisions
  • Fixed all agents and IP addresses to have null strings (some were missed in the last update which could lead to problems until you ran the script again).
  • Fixed bug which kept saying bugs were updated that were already updated.
  • Fixed bug where default operation would not get options processed unless the Operation option was specified
  • Fixed phpBB session errors caused by outputting HTML before session management started
  • Fixed debug messages when $_GET[...] parameter wasn't specified (thanks doktornotor and AmigoJack)
  • Changed get_parameter function to use request_var for security (thanks, Paul)
  • Fixed bug where updating bots didn't write out delete summary
  • Stopped updating bots if help requested
  • Moved fix_bots script function into update_bots just in case somebody didn't run it
  • Changed default level to 50 to get all visiting bots
  • Added $debug_level to output debugging strings instead of commenting them out
  • Added Debug option to control $debug_level from the command line
  • Various updates to list of bots to add and update
  • Improved the documentation comments
  • Added donation information to output
See the code's header comments for a full change list. More details about the changes in this version are in the announcement post.

For help after uploading the script, type something like http://example.com/phpbb3/manage_bots.php?? (note the two question marks!).

The new manage_bots script is in the attached file.

Let me know what you think and PM me if you find any bugs.

Steve
Attachments
manage_bots.php
Manage_Bots 5.5 PHP script
(144.01 KiB) Downloaded 803 times
Last edited by Pony99CA on Sun May 27, 2012 8:07 am, edited 17 times in total.
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: No registered users and 22 guests