[BETA]Search Bot Indexing MOD v1.3.1

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Post Reply
User avatar
Raimon
Former Team Member
Posts: 12088
Joined: Tue May 30, 2006 5:31 pm
Location: Netherlands
Name: Raimon Meuldijk
Contact:

[BETA]Search Bot Indexing MOD v1.3.1

Post by Raimon » Sun Nov 19, 2006 5:02 pm

Image


Search Bot Indexing MOD v1.3.0 (Bot = search engine spider, not spam bot, to avoid any confusion.)

Hello all, this MOD was originally written by Adam Marcus (addit), chris3471 and with no inaktivitie of the orginal autour i have take his develpment over.

The MOD is stable to the best of my knowledge, I've done a few test installs to make sure it works. I've had it running for about a week on my forum with no problems. I just figured I should put in back in BETA to make sure there were no problems before submitting it to the MOD db.

This mod allows your site to be indexed by removing SIDs, however it does not by default optimise your site to be indexed. It does however log bot visits and allow you to create custom made styles for certain bots.

These are the original features features where Addit left it:

- A brand spanking new admin page! - "Manage Bots"
- A schweet installation script if I do say so myself.
- Bots can index your forums as SIDs are removed.
- Bots time of last visit are recorded.
- Bots are detected by both user agent and user ip. This limits the possibility of users masquerading around as the googlebot having there SIDs removed.
- Users/bots that match some of the criteria but not all are pulled into a pending bots section where you can decide their fate next time they show up - do you want them to be classed as a bot?
- A list is retained of the past five times each bot has visited your site.
- Multiple language support. However as of yet no one has chosen to utilize this (write language packs).
- A record of how many times each bot has visted your forums.
- A record of how many pages each bot looked at last time they visted your site. With this, an estimated percentage of how much of your forums they looked at!
- Bots are counted as visitors and appear in "whos online" lists etc.
- The ability to show different bots different templates/styles of your site.
-Show Bots by name in the who's online in the admin index and the forum index.
-Show Bots by name on view forum pages

Features currently being worked on:

- More bots with difrent ips on the same bot name on viewonline table.
- The colour to be choising of the bot for viewonline tabel en viewonline page.

If you have a new feature request feel free to PM me or post in this thread.

This is from the orginal treat that cane you find over Here
The first topic: here

For a live demo check out Here
Few live image screens:
Image
Currently a Bot online

Image
A look in the administration panel when where a bot have visit your board

Image
A look at the viewonline.php page


Download can you found now Here
Cvs can be found here
Now for what for function you like to seeing let me know for this mod,
I have now the devoplment taking over for this mod from the original autour.

Happy testing Raimon

:)

Mod History:

Project from version 1.3.0 taking over
Added smal fix, and to input more ips of bots version 1.3.1

Changelog:
10-4-2007 Demo link change after a huge server crash with my old hoster
Little start logo added
Last edited by Raimon on Sun Sep 16, 2007 5:18 pm, edited 8 times in total.
Need phpBB installation, extenstions, Styles or integrate phpBB with you website?
Contact me for fair prices and good service!

User avatar
bonelifer
Community Team Member
Community Team Member
Posts: 3463
Joined: Wed Oct 27, 2004 11:35 pm
Name: William
Contact:

Post by bonelifer » Sun Nov 19, 2006 11:05 pm

The obvious improvement to this mod is some sort of check boxes at the bottom in the confirm page. It's annoying to have to click on the bottom ADD, then the popup page OK, the scroll down do it all over again for just one IP for the SAME bot. So having Delete/Add check boxes and a submit button at the bottom would be nice. Currently the Inktomi bot decided all of a sudden to send all it's spiders to my site. I haven't been able to clear the page in over a MONTH due to their being way to many spiders that came all at once. This has therefore slowed down the loading of that page as I no longer try to add any bot IPs as it's just become more of a hassle than it's worth the way it's set up right now to work.
Knowledge Base | phpBB Board Rules | Search Customisation Database
Image
Please don't contact me via PM or email for phpBB support .

User avatar
igorw
Former Team Member
Posts: 8024
Joined: Fri Dec 16, 2005 12:23 pm
Location: {postrow.POSTER_FROM}
Name: Igor Wiedler

Post by igorw » Mon Nov 20, 2006 12:05 am

Looks very nice! I'm watching this :)
Igor Wiedler | area51 | GitHub | trashbin | Formerly known as evil less than three

Peter77sx
Registered User
Posts: 3253
Joined: Wed Nov 09, 2005 2:51 pm

Post by Peter77sx » Mon Nov 20, 2006 1:09 am

Features currently being worked on:

- More bots with difrent ips on the same bot name on viewonline table.


Like maybe GoogleBot ( 3 ), MSN ( 2 ) ?

killyouall
Registered User
Posts: 523
Joined: Tue Oct 18, 2005 3:04 am
Contact:

Post by killyouall » Mon Nov 20, 2006 3:02 am

this compadible wit easymod? if not you should try to make it lol

super-dave
Registered User
Posts: 74
Joined: Thu Oct 27, 2005 2:05 pm
Contact:

Post by super-dave » Mon Nov 20, 2006 1:40 pm

killyouall wrote: this compadible wit easymod? if not you should try to make it lol


i installed this mod today and it IS compatible with easymod. i ran the bot_install.php file, then i ran the mod install through easymod and it worked like a charm.

actually, i have another question with compatibility.

i installed this mod (as stated) with no problems. however, i came across a small problem with another mod:

in my includes/sessions.php file, i ended up with this line:
if ( !empty($SID) && !preg_match('#sid=#', $url) && !IS_ROBOT && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Googlebot') && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'slurp@inktomi.com;'))


everything before && !IS_ROBOT is fine, but the google bot mod (remove sids, allowing google to index my site) wanted to add the rest.

is this line right? i'm not that fluent in php to know whether both mods will work with this line as is, or if i need to modify it.

i've only had your mod up for a few hours, so it's not showing any bot hits yet.

how often do bots usually hit? my awstats shows that google bot visits my sites regularly! :)

super-dave
Registered User
Posts: 74
Joined: Thu Oct 27, 2005 2:05 pm
Contact:

Post by super-dave » Mon Nov 20, 2006 1:51 pm

actually, i re-read your first post and it said that this mod helps bots access and index my site. so i shouldn't need the extra mod.

i backed up my sessions.php file and got rid of the extra mod.

so hopefully it will work ok now.

just gotta wait and see if any bots are registered. :)

User avatar
DarthSpader
Registered User
Posts: 184
Joined: Tue Sep 30, 2003 1:56 pm

Post by DarthSpader » Tue Nov 21, 2006 2:05 am

How could you code this so that the Visits count of a SearchBot (Google, for example) would be displayed on the index?
I'm NOT Your Father Either.

Xorro
Registered User
Posts: 62
Joined: Wed Jan 04, 2006 1:24 pm
Location: UK

Post by Xorro » Tue Nov 21, 2006 9:55 am

I installed the original version of this mod and it works fine.

It would be great if the code would allow for the bots to be a different colour from other users. I have manually edited the code on my site, but if this could be soft-coded in someway it would be useful.

I agree with the count of a bot, I often get Googlebot logged in 20 times on my site.

Example of the alternative colours -
Image
The UK Hyundai Coupe Site and forums
Pure MPS Mazda MPS owners forum

emexci
Registered User
Posts: 15
Joined: Sat Dec 24, 2005 5:19 pm
Contact:

Thanks

Post by emexci » Tue Nov 21, 2006 11:31 am

nice mod, thanks

super-dave
Registered User
Posts: 74
Joined: Thu Oct 27, 2005 2:05 pm
Contact:

Post by super-dave » Wed Nov 22, 2006 7:00 am

found a possible bug.

it seems that there's a limit to the number of ip addresses that can be added to a bot. i'm assuming that this is an issue within the structure of the database entry concerning ips.

i've tried adding ips manually and once the string gets long, it won't allow more to be added.

are there limits to the length of these entries in the database?

if so, perhaps the bots can be broken up into categories. rather than just google bot, it could be google/mediapartners bot, google mozilla bot, etc.

just a thought :)

ShoKKer
Registered User
Posts: 2
Joined: Sun Apr 17, 2005 5:05 pm
Contact:

Post by ShoKKer » Wed Nov 22, 2006 11:17 pm

I also think it would be great if this could be integrated with the AUC mod from Austin to allow for different colors for different Bots.
It might even be easier just to give each bot a hardcoded color and use the AUC to identify them manually in admin, but different colors would be nice!

I would also like to see check boxes and a Delete/Add button added to the admin screen to allow for easy deletion of multiple false bots or adding multiple NEW bots.
I have several very large sites that get multiple hits and are constantly being spidered. Sometimes I have to add/delete large amounts per week. This would make it much easier to add or delete them.

I am also a willing and experienced Beta Tester and I do have several dedicated test sites.

Thanks for your time and the great mod!

super-dave
Registered User
Posts: 74
Joined: Thu Oct 27, 2005 2:05 pm
Contact:

Post by super-dave » Thu Nov 23, 2006 1:52 am

anyone got any advice about how to know whether i should add or ignore bots?

here's some of the ones i'm getting:
Googlebot 125.241.109.122 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.158.50 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.189.210 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.241.167.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.241.8.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.195.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.101.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.143.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.132.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.245.61 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.169.3 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.196.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.125.162 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add


and strangely enough, i only get googlebot visiting my site. i've never had msn or yahoo. but this mod has only been installed for a week. :)

super-dave
Registered User
Posts: 74
Joined: Thu Oct 27, 2005 2:05 pm
Contact:

Post by super-dave » Thu Nov 23, 2006 1:56 am

Xorro wrote: I installed the original version of this mod and it works fine.

I have manually edited the code on my site, but if this could be soft-coded in someway it would be useful.


would you mind pasting the code you used to do this?

damianconnolly
Registered User
Posts: 9
Joined: Mon Dec 20, 2004 11:42 am
Contact:

Post by damianconnolly » Fri Nov 24, 2006 4:05 pm

I installed this mod about a week ago, and I'm having a wierd problem with it now....

I had to do a restore of the php/html etc from shortly after the mod was done because another mod made the board unstable, and now when I go into the "Manage Bots" page in Adminisrtation, I just get

Code: Select all

Couldn't query bots.

DEBUG MODE

SQL Error : 1146 Table 'damianconn.BOTS_TABLE' doesn't exist

SELECT bot_id, bot_name, last_visit, bot_visits, bot_pages FROM BOTS_TABLE

Line : 507
File : admin_bots.php
Any ideas :oops:

Post Reply

Return to “[2.0.x] MODs in Development”