[BETA]Search Bot Indexing MOD v1.3.1

Raimon · Post by **Raimon** » Sun Nov 19, 2006 5:02 pm

Search Bot Indexing MOD v1.3.0 (Bot = search engine spider, not spam bot, to avoid any confusion.)

Hello all, this MOD was originally written by Adam Marcus (addit), chris3471 and with no inaktivitie of the orginal autour i have take his develpment over.

The MOD is stable to the best of my knowledge, I've done a few test installs to make sure it works. I've had it running for about a week on my forum with no problems. I just figured I should put in back in BETA to make sure there were no problems before submitting it to the MOD db.

This mod allows your site to be indexed by removing SIDs, however it does not by default optimise your site to be indexed. It does however log bot visits and allow you to create custom made styles for certain bots.

These are the original features features where Addit left it:

- A brand spanking new admin page! - "Manage Bots"
- A schweet installation script if I do say so myself.
- Bots can index your forums as SIDs are removed.
- Bots time of last visit are recorded.
- Bots are detected by both user agent and user ip. This limits the possibility of users masquerading around as the googlebot having there SIDs removed.
- Users/bots that match some of the criteria but not all are pulled into a pending bots section where you can decide their fate next time they show up - do you want them to be classed as a bot?
- A list is retained of the past five times each bot has visited your site.
- Multiple language support. However as of yet no one has chosen to utilize this (write language packs).
- A record of how many times each bot has visted your forums.
- A record of how many pages each bot looked at last time they visted your site. With this, an estimated percentage of how much of your forums they looked at!
- Bots are counted as visitors and appear in "whos online" lists etc.
- The ability to show different bots different templates/styles of your site.
-Show Bots by name in the who's online in the admin index and the forum index.
-Show Bots by name on view forum pages

Features currently being worked on:

- More bots with difrent ips on the same bot name on viewonline table.
- The colour to be choising of the bot for viewonline tabel en viewonline page.

If you have a new feature request feel free to PM me or post in this thread.

This is from the orginal treat that cane you find over Here
The first topic: here

For a live demo check out Here
Few live image screens:

Currently a Bot online

A look in the administration panel when where a bot have visit your board

A look at the viewonline.php page

Download can you found now Here
Cvs can be found here
Now for what for function you like to seeing let me know for this mod,
I have now the devoplment taking over for this mod from the original autour.

Happy testing Raimon

Mod History:

Project from version 1.3.0 taking over
Added smal fix, and to input more ips of bots version 1.3.1

Changelog:
10-4-2007 Demo link change after a huge server crash with my old hoster
Little start logo added

Post by **bonelifer** » Sun Nov 19, 2006 11:05 pm

The obvious improvement to this mod is some sort of check boxes at the bottom in the confirm page. It's annoying to have to click on the bottom ADD, then the popup page OK, the scroll down do it all over again for just one IP for the SAME bot. So having Delete/Add check boxes and a submit button at the bottom would be nice. Currently the Inktomi bot decided all of a sudden to send all it's spiders to my site. I haven't been able to clear the page in over a MONTH due to their being way to many spiders that came all at once. This has therefore slowed down the loading of that page as I no longer try to add any bot IPs as it's just become more of a hassle than it's worth the way it's set up right now to work.

igorw · Post by **igorw** » Mon Nov 20, 2006 12:05 am

Looks very nice! I'm watching this

Peter77sx · Post by **Peter77sx** » Mon Nov 20, 2006 1:09 am

Features currently being worked on:

- More bots with difrent ips on the same bot name on viewonline table.

Like maybe GoogleBot ( 3 ), MSN ( 2 ) ?

killyouall · Post by **killyouall** » Mon Nov 20, 2006 3:02 am

this compadible wit easymod? if not you should try to make it lol

super-dave · Post by **super-dave** » Mon Nov 20, 2006 1:40 pm

killyouall wrote: this compadible wit easymod? if not you should try to make it lol

i installed this mod today and it IS compatible with easymod. i ran the bot_install.php file, then i ran the mod install through easymod and it worked like a charm.

actually, i have another question with compatibility.

i installed this mod (as stated) with no problems. however, i came across a small problem with another mod:

in my includes/sessions.php file, i ended up with this line:

if ( !empty($SID) && !preg_match('#sid=#', $url) && !IS_ROBOT && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Googlebot') && !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'slurp@inktomi.com;'))

everything before && !IS_ROBOT is fine, but the google bot mod (remove sids, allowing google to index my site) wanted to add the rest.

is this line right? i'm not that fluent in php to know whether both mods will work with this line as is, or if i need to modify it.

i've only had your mod up for a few hours, so it's not showing any bot hits yet.

how often do bots usually hit? my awstats shows that google bot visits my sites regularly!

super-dave · Post by **super-dave** » Mon Nov 20, 2006 1:51 pm

actually, i re-read your first post and it said that this mod helps bots access and index my site. so i shouldn't need the extra mod.

i backed up my sessions.php file and got rid of the extra mod.

so hopefully it will work ok now.

just gotta wait and see if any bots are registered.

DarthSpader · Post by **DarthSpader** » Tue Nov 21, 2006 2:05 am

How could you code this so that the Visits count of a SearchBot (Google, for example) would be displayed on the index?

Xorro · Post by **Xorro** » Tue Nov 21, 2006 9:55 am

I installed the original version of this mod and it works fine.

It would be great if the code would allow for the bots to be a different colour from other users. I have manually edited the code on my site, but if this could be soft-coded in someway it would be useful.

I agree with the count of a bot, I often get Googlebot logged in 20 times on my site.

Example of the alternative colours -

emexci · Post by **emexci** » Tue Nov 21, 2006 11:31 am

nice mod, thanks

super-dave · Post by **super-dave** » Wed Nov 22, 2006 7:00 am

found a possible bug.

it seems that there's a limit to the number of ip addresses that can be added to a bot. i'm assuming that this is an issue within the structure of the database entry concerning ips.

i've tried adding ips manually and once the string gets long, it won't allow more to be added.

are there limits to the length of these entries in the database?

if so, perhaps the bots can be broken up into categories. rather than just google bot, it could be google/mediapartners bot, google mozilla bot, etc.

just a thought

ShoKKer · Post by **ShoKKer** » Wed Nov 22, 2006 11:17 pm

I also think it would be great if this could be integrated with the AUC mod from Austin to allow for different colors for different Bots.
It might even be easier just to give each bot a hardcoded color and use the AUC to identify them manually in admin, but different colors would be nice!

I would also like to see check boxes and a Delete/Add button added to the admin screen to allow for easy deletion of multiple false bots or adding multiple NEW bots.
I have several very large sites that get multiple hits and are constantly being spidered. Sometimes I have to add/delete large amounts per week. This would make it much easier to add or delete them.

I am also a willing and experienced Beta Tester and I do have several dedicated test sites.

Thanks for your time and the great mod!

super-dave · Post by **super-dave** » Thu Nov 23, 2006 1:52 am

anyone got any advice about how to know whether i should add or ignore bots?

here's some of the ones i'm getting:

Googlebot 125.241.109.122 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.158.50 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.189.210 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.241.167.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.241.8.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.195.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.101.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.143.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.132.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.245.61 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 124.1.169.3 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.196.130 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add
Googlebot 125.240.125.162 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Ignore Add

and strangely enough, i only get googlebot visiting my site. i've never had msn or yahoo. but this mod has only been installed for a week.

super-dave · Post by **super-dave** » Thu Nov 23, 2006 1:56 am

Xorro wrote: I installed the original version of this mod and it works fine.

I have manually edited the code on my site, but if this could be soft-coded in someway it would be useful.

would you mind pasting the code you used to do this?

damianconnolly · Post by **damianconnolly** » Fri Nov 24, 2006 4:05 pm

I installed this mod about a week ago, and I'm having a wierd problem with it now....

I had to do a restore of the php/html etc from shortly after the mod was done because another mod made the board unstable, and now when I go into the "Manage Bots" page in Adminisrtation, I just get

Code: Select all

Couldn't query bots.

DEBUG MODE

SQL Error : 1146 Table 'damianconn.BOTS_TABLE' doesn't exist

SELECT bot_id, bot_name, last_visit, bot_visits, bot_pages FROM BOTS_TABLE

Line : 507
File : admin_bots.php

Any ideas