Spiders & bots to add to phpBB

[ToonArmy]

http://www.ip2location.com/65.55.109.155

I have had a visitor from that IP. The ISP is Microsoft, but it's from NY (MS is in Redmond). Is that a bot or does MS run an ISP?

Do you suppose that the database is incorrect? That's much more likely.

Why don't you look up Microsoft's netblock? See what that gives. Much more accurate results, no doubt!

If the site's not correct, then can you recommend a better one? I have been using it for a while and most of the time it gives good results...

The IP appears to be in a NYC data centre, just because Microsoft HQ is in Redmond that doesn't mean everything they do is. Don't put all your eggs in one basket and all that.

[MPG187]

So what would an NYC data center computer be doing browsing my site?

[Pony99CA]

What I mean in the message above is that after I ran add_bots.php there was no bot displayed on the section "Who is online". Only registered users are displayed there.

Unless you've changed something, bots are considered registered and displayed in the WHO IS ONLINE section. Look at the phpBB.com page and you'll likely see a few bots listed there.

But I must say there are a lot of bots visiting my forum, and I see those bots as visitors (logged as administrator I can see visitors agent and some of those visitors had msn, googlebot, yahoo! slurp, etc agents.)

Where are you seeing those agents? If you're clicking on WHO IS ONLINE as an admin, the agents show up there (as I mentioned previously).

Please use the exact names of pages and fields (using the default language pack) when reporting things. Making people guess what you mean just delays things.

This was yesterday. Today still no bots on "who is online".

Again, you'll only see them listed if they're currently online or were online within the time specified by the setting that I pointed out. If your board isn't very well-known, bots might not hang out there for long periods of time.

PS: I deleted now all bots and I added some of them manualy and it is working now.
Maybe I will add the entire list without '52', '1', '999' and I guess this way will work.

I'm not sure what "without '52', '1', '999'" means. There's only one thing that you need to do to change the level of bots added (either specify the &level= parameter on the call or change the $default_level variable).

If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)

Steve

[Pony99CA]

... If you find one, please post the details here (see previous posts for an idea of how they look).

And a bot that I haven't saw on your list but visited my forum:
agent: Mozilla/5.0+(compatible;+OkidokiBot/0.1;++http://www.okidoki.ro/bot.html)

Thanks. I'll add that to the list when I get time. (Are you in Romania, by any chance?)

Steve

[Pony99CA]

So what would an NYC data center computer be doing browsing my site?

It could be an employee who works at some Micorosoft office in New York.

Please, let's not hijack this thread to guess why various IP addresses may show up on your site. Most bots will present an obvious agent string, and those are the ones that we want to capture.

Steve

[MPG187]

So what would an NYC data center computer be doing browsing my site?

It could be an employee who works at some Micorosoft office in New York.

Please, let's not hijack this thread to guess why various IP addresses may show up on your site. Most bots will present an obvious agent string, and those are the ones that we want to capture.

Steve

When I posted I thought they mighta been a bot. It guess MS Employees love my site. Or maybe they don't like the modded Xbox stuff. Anyways, I'll end it here...

[onehundredandtwo]

I have had MSNBot view my website with an Internet Explorer user-agent before, and I doubt anyone from Microsoft would have found that website. My best guess is that Microsoft has some type of bot that checks if websites suspected of spamming / new websites are serving different content for bots.

[ToonArmy]

So what would an NYC data center computer be doing browsing my site?

Indexing it? It's a MSN bot IP: 155.109.55.65.in-addr.arpa domain name pointer msnbot-65-55-109-155.search.msn.com.

[clava]

If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)

Steve

Hello,
I am very sory I could not make myself understood.
My english it is not so good.
Anyway, I have sent you a PM with detailed actions I made on my bot list and a link to my working forum with a lot of bots displayed on "Who is online".

[MPG187]

Saying that it's an MSN bot does make sense. But it doesn't show up as the MSN bot.

Unless is MSN using another IP address, one that is not registered with phpbb yet?

Next time the MSN bot it on my site I'll have to check it's user agent. I do remember when that visitor was on my site it was using IE...

EDIT (Not Double Posting)

Guest
IP: 75.101.184.216 » Whois
grvcrawler/0.3

[Pony99CA]

If you think there's a bug in the script, please provide me exact details of where you got it, what (if any) edits you made (or E-mail me the script that you used) and how you invoked the script. (Send the details via PM, please; don't post them here.)

Anyway, I have sent you a PM with detailed actions I made on my bot list and a link to my working forum with a lot of bots displayed on "Who is online".

After PM discussion with clava, he mentioned that he is using the following MODs (among others):

• Override_the_edit_time_limit-1-0-1
• top_posters_1_0_0a
• users_online_time_102
• guests_online_past_24_hours
• activity_stats_mod

I don't know those MODs, but I suspect there could be some interaction with some of those. If anybody using those has run the add_bots script, have you noticed any problems?

Thanks,
Steve

[Pony99CA]

Saying that it's an MSN bot does make sense. But it doesn't show up as the MSN bot.

Unless is MSN using another IP address, one that is not registered with phpbb yet?

Next time the MSN bot it on my site I'll have to check it's user agent. I do remember when that visitor was on my site it was using IE...

EDIT (Not Double Posting)

Guest
IP: 75.101.184.216 » Whois
grvcrawler/0.3

That's not from MSN; that's the Amazon Electric Cloud bot that I reported on February 8. It's got a completely different IP address, but it resolves to the Electric Cloud nonetheless.

Steve

[ToonArmy]

That's not from MSN; that's the Amazon Electric Cloud bot that I reported on February 8. It's got a completely different IP address, but it resolves to the Electric Cloud nonetheless.

Those were two posts in one.

[maxwell2]

How do I add a bot to get a message that such a bot already exists
How to remove all the bots and add new ones? Modification is to add mass bots?

[Pony99CA]

manage_bots has been updated to version 5.5. manage_bots is basically version 4.0 of add_bots with the functions of delete_bots and list_bots merged in. This makes things simpler for me as I don't have to keep the bots tables in sync and simpler for you because there's only one script to worry about.

To install and use manage_bots, please read the comments at the top of the script.

New features and changes over the original add_bots script include the following (new features are in bold):

• Combined add_bots, delete_bots and list_bots into one manage_bots script
• Adding bots now only shows the bots added unless you specify the ListAll option.
• Added the update operation to just invoke the update_bots function
• Added new command to check for conflicting bots
• Added Help (or "?") command to display command help; the "?" works even when not logged in to help you get started
• Added credits, version number and link to current script to help
• Updated security to only allow board administrators with manage bots permission to run this script.
• Added ability to set bot's active status and last visit timestamp. Combined with the List enhancements below, you can now use Delete without losing your Bot visit data.
• Added ability to set bot's E-mail address and Web site (based on an idea by AmigoJack). This is only viewable in the ACP user view (unless you create a MOD to allow viewing Bot profiles).
• Updated List command to dump bots' new fields listed above (doing a List before a Delete allows restoring that data after the delete).
• Added optional formatting for List output and added format and number options
• Added Activate and Deactivate commands to change bots activation status quickly.
• Added password support to prevent unauthorized users from running the script
• Added warning if conflicting agent strings or IP addresses are found
• Moved non-debugging strings into array in customization section suitable for translation
• Added command and parameter names to language array to allow translating them
• Added checks for maximum values of some parameters
• Cleared the bot cache so results take effect immediately (I hope)
• Fixed bug in updating if new bot name already existed
• Moved checks to see if the bot was in the user table to the bot_defined function
• Used clean username for add and delete comparisons to avoid collisions
• Fixed all agents and IP addresses to have null strings (some were missed in the last update which could lead to problems until you ran the script again).
• Fixed bug which kept saying bugs were updated that were already updated.
• Fixed bug where default operation would not get options processed unless the Operation option was specified
• Fixed phpBB session errors caused by outputting HTML before session management started
• Fixed debug messages when $_GET[...] parameter wasn't specified (thanks doktornotor and AmigoJack)
• Changed get_parameter function to use request_var for security (thanks, Paul)
• Fixed bug where updating bots didn't write out delete summary
• Stopped updating bots if help requested
• Moved fix_bots script function into update_bots just in case somebody didn't run it
• Changed default level to 50 to get all visiting bots
• Added $debug_level to output debugging strings instead of commenting them out
• Added Debug option to control $debug_level from the command line
• Various updates to list of bots to add and update
• Improved the documentation comments
• Added donation information to output

See the code's header comments for a full change list. More details about the changes in this version are in the announcement post.

For help after uploading the script, type something like http://example.com/phpbb3/manage_bots.php?? (note the two question marks!).

The new manage_bots script is in the attached file.

Let me know what you think and PM me if you find any bugs.

Steve