[BETA]Search Bot Indexing MOD v1.3.0 (was Bot Indexing MOD)

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
Post Reply
chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

[BETA]Search Bot Indexing MOD v1.3.0 (was Bot Indexing MOD)

Post by chris3471 » Tue Jun 06, 2006 6:45 pm

Search Bot Indexing MOD v1.3.0 (Bot = search engine spider, not spam bot, to avoid any confusion.)

Hello all, this MOD was originally written by Adam Marcus (addit) and with his permission I've taken over the developement, I've made a number of changes and have a few more in the works. I'd like to get a few beta testers to give this a try.

The MOD is stable to the best of my knowledge, I've done a few test installs to make sure it works. I've had it running for about a week on my forum with no problems. I just figured I should put in back in BETA to make sure there were no problems before submitting it to the MOD db.

The show bots in who's online feature will detect the body_link color and will show the bots name as the same color as the regular users for most themes. I've made a forth font color MOD (link in sig) in case anyone wants a differen't unique color for the bots.


This mod allows your site to be indexed by removing SIDs, however it does not by default optimise your site to be indexed. It does however log bot visits and allow you to create custom made styles for certain bots.

These are the original features features where Addit left it:

- A brand spanking new admin page! - "Manage Bots"
- A schweet installation script if I do say so myself.
- Bots can index your forums as SIDs are removed.
- Bots time of last visit are recorded.
- Bots are detected by both user agent and user ip. This limits the possibility of users masquerading around as the googlebot having there SIDs removed.
- Users/bots that match some of the criteria but not all are pulled into a pending bots section where you can decide their fate next time they show up - do you want them to be classed as a bot?
- A list is retained of the past five times each bot has visited your site.
- Multiple language support. However as of yet no one has chosen to utilize this (write language packs).
- A record of how many times each bot has visted your forums.
- A record of how many pages each bot looked at last time they visted your site. With this, an estimated percentage of how much of your forums they looked at!
- Bots are counted as visitors and appear in "whos online" lists etc.
- The ability to show different bots different templates/styles of your site.

Added features are since I took over the developement:
-Show Bots by name in the who's online in the admin index and the forum index.
-Show Bots by name on view forum pages

Known Bugs Fixed:
- Can no longer add your own details as a bot since it would consquently lock you out from the admin panel.
- Two bots can be no longer be given the same bot name.

Features currently being worked on:

-Add pagination to the manage bots page.
-Add an all time pageview count (currently the bots pageview number resets every 24 hours) all time pageview count will show hits for all time.

If you have a new feature request feel free to PM me or post in this thread.


SCREENSHOTS AND DOWNLOAD HERE, UPDATE FROM 1.2.0 IS IN THE DOWNLOAD FILE ALSO!!!

*EDIT* Now compatible with EasyMOD, you do still have to run the installer, bot_install.php though, it will be in your phpbb root folder after you run EasyMOD.

Here's the original thread started by addit if anyone wants it for a reference.
Last edited by chris3471 on Wed Jun 07, 2006 3:46 pm, edited 5 times in total.

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 1:20 am

Oops, I made a small mistake, bots have admin link color.

I've already fixed it in the download file for anyone that hasn't already downloaded it.

Here's the fix.

Code: Select all

#
#-----[ OPEN ]--------------------------------------------
#
 
includes/page_header.php

#
#-----[ FIND ]--------------------------------------------
#

$row['is_robot'] = '<span style="color:#' . $theme['fontcolor3'] . '"><b>' . $row['is_robot'] . '</b></span>';
 
#
#-----[ IN-LINE FIND ]--------------------------------------------
#

$theme['fontcolor3'] 

#
#-----[ REPLACE WITH ]--------------------------------------------
#

$theme['body_link'] 

#
#-----[ SAVE/CLOSE ALL FILES ]--------------------------------------------
#
#EOM

Last edited by chris3471 on Wed Jun 07, 2006 3:08 am, edited 1 time in total.

Peter77sx
Registered User
Posts: 3253
Joined: Wed Nov 09, 2005 2:51 pm

Post by Peter77sx » Wed Jun 07, 2006 2:13 am

Hello, is it possible to skip the sessions ID part of this MOD and still have it work? I already have the SIDs taken care of.

thanks.

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 2:35 am

No I don't think it would work without that part, the script calls to sessions for several functions.

I do know that the MOD is compatible with Craven de Kere's SEO MOD Able2Know.com SEO 2.0.0, as I have both installed with no problems.

The session ids are assigned to the bots for the purpose of identifying them only it has no effect really on SEO which I assume is what you mean by having the sid taken care of..

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 5:13 am

Just a note, I made a few changes to the INSTALL.txt file and the MOD is now compatible with EasyMOD. However, you do still have to run the bot_install.php file the path after running EasyMOD will be yoursite.com/your-phpbb-install/bot_install.php in other words it will be in your forum root directory just go to it with your browser.

Xusqui
Registered User
Posts: 84
Joined: Wed Jun 02, 2004 10:35 pm
Contact:

Post by Xusqui » Wed Jun 07, 2006 10:31 am

NICE!!!!

I have the latest bot indexing mod in my forum... I'll keep on trying this one!!!

Thnx indeed!!!!

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 12:17 pm

No problem, I'm happy to be of assistance. :D

Xusqui
Registered User
Posts: 84
Joined: Wed Jun 02, 2004 10:35 pm
Contact:

Post by Xusqui » Wed Jun 07, 2006 12:38 pm

Well... I completed uninstalled the previous version (but the tables in the db) and re-installed the new one...

1st issue:
The spiders lines at the bottom of the forum are not displayed... I mean, below the users online (now shows the name of the bots, that's cool) there is a code (inside the index_body.tpl) that should display accurate information about every bot connected (am I righ?) But that info is not shown in my case.

Shouldn't you create a "spider_user_row" section in the template assignation variables seccion inside index.php? You know what I men (don't xou?)

Thank u!!
Last edited by Xusqui on Wed Jun 07, 2006 12:48 pm, edited 1 time in total.

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 12:45 pm

It's normal the first time a spider comes around to display as a guest, this is for each IP that a spider may use. They get placed in a pending table first so you can decide if you want them listed or not. If you go to the Manage Bots page in the ACP you will probably see a bunch of them in the pending table, just add them all and it will work as it's supposed to.

You mean the admin/index.tpl right?

You did replace all the files with the ones in the zip right?

Also you may want to drop the bots tables in the db, and also the is_robot table in phpbb_sessions and try the installer if it still doesn't work.

There's actually no change in the database so that is optional if you're just upgrading from 1.20 but the new installer will give you almost every ip for all the major search engines.

Xusqui
Registered User
Posts: 84
Joined: Wed Jun 02, 2004 10:35 pm
Contact:

Post by Xusqui » Wed Jun 07, 2006 12:54 pm

well... Not exactly... I understand what you say, but as I didn't clear the table, the deffinitions of my robots are still there, so I don't see no robots to add, but that's not the question

I mean, in the index_body.tpl you got to insert this code:

Code: Select all

<!-- BEGIN spider_user_row -->
  <tr> 
	<td width="20%" class="{spider_user_row.ROW_CLASS}">&nbsp;<span class="gen">{spider_user_row.USERNAME}</span>&nbsp;</td>
	<td width="20%" align="center" class="{spider_user_row.ROW_CLASS}">&nbsp;<span class="gen">{spider_user_row.STARTED}</span>&nbsp;</td>
	<td width="20%" align="center" nowrap="nowrap" class="{spider_user_row.ROW_CLASS}">&nbsp;<span class="gen">{spider_user_row.LASTUPDATE}</span>&nbsp;</td>
	<td width="20%" class="{spider_user_row.ROW_CLASS}">&nbsp;<span class="gen"><a href="{spider_user_row.U_FORUM_LOCATION}" class="gen">{spider_user_row.FORUM_LOCATION}</a></span>&nbsp;</td>
	<td width="20%" class="{spider_user_row.ROW_CLASS}">&nbsp;<span class="gen"><a href="{spider_user_row.U_WHOIS_IP}" target="_phpbbwhois">{spider_user_row.IP_ADDRESS}</a></span>&nbsp;</td>
  </tr>
  <!-- END spider_user_row -->
  <tr> 
	<td colspan="5" height="1" class="row3"><img src="../../../../phpBB2/templates/subSilver/templates/subSilver/images/spacer.gif" width="1" height="1" alt="."></td>
  </tr>
1.- That will give you a 404 error in your log when trying to search the imaged stated in this line:

Code: Select all

"><img src="../../../../phpBB2/templates/subSilver/templates/subSilver/images/spacer.gif" width="1" height="1" alt=".">
2.- I think this part of the code is to display accurate information about the robot, when it logged int, and so on, isn't it? If I'm true, that info isn't shown at my forum... That's what I meant

Thanx again

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 12:56 pm

Oh, no that image is just a spacer in between the rows, between registered user row, guest user row and spider user row. It has no function but to leave a space between the rows.
Last edited by chris3471 on Wed Jun 07, 2006 12:59 pm, edited 1 time in total.

Xusqui
Registered User
Posts: 84
Joined: Wed Jun 02, 2004 10:35 pm
Contact:

Post by Xusqui » Wed Jun 07, 2006 12:58 pm

Oh... My God!!!!!!!!

How stupid I am...

I just changed the wrong file... :oops:

I missunderstood :oops: I changed index_body.tpl instead of admin/index_body.tpl

Now everything is clear...

Sorry for beeing so silly!!!!

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 1:01 pm

It's ok, I spent 3 hours trying to figure out why something wouldn't work the other day only to find out I had changed my theme and wasn't using subSilver.

Oh just a note too, it's normal for multiple bots to show in the admin index with the same ip, sometimes they are viewing more than one thread at a time.

I'm trying to make it only show once on the forum index but instead it will show once per ip, but even as I write this I think it just occured to me why.

Xusqui
Registered User
Posts: 84
Joined: Wed Jun 02, 2004 10:35 pm
Contact:

Post by Xusqui » Wed Jun 07, 2006 1:13 pm

Ok... Thnx very much... I really feel sorry...

A good feature would be that you could decide, via the ACP the color of the displayed bot...

and... Of course, you won't get a 404 error, but the error log will show you that request as a 404 error, that's what I ment!!

chris3471
Registered User
Posts: 157
Joined: Wed Dec 07, 2005 8:32 pm

Post by chris3471 » Wed Jun 07, 2006 1:19 pm

I'm working on a fourth font color MOD so that you will be able to make the bots a differen't color.

I'll look into that 404 too thanks.

*edit* I'm not seeing the 404 in my error log, are you using subSilver? If not your theme may not have the spacer image.

Or perhaps the error was because it was looking for it in the wrong directory when you had the code in the wrong template, that's possibly the answer.

Post Reply

Return to “[2.0.x] MODs in Development”