[INFO] How gogle PHPBB!

A place for MOD Authors to post and receive feedback on MODs still in development. No MODs within this forum should be used within a live environment! No new topics are allowed in this forum.
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: MOD Development Forum rules

On February 1, 2009 this forum will be set to read only as part of retiring of phpBB2.
User avatar
cdkrg
Registered User
Posts: 706
Joined: Fri Jul 12, 2002 12:35 pm
Contact:

Post by cdkrg »

R. U. Serious wrote: 1. How are you going to "hide" the links? -> You really can't, only make them less obvious.
2a. Do you want to duplicate each link? -> confusion & pagesize
2b. Only duplicate one link on the index? -> every spidered page will be on level deeper and possibly have a worse rank. Still confusion if normal users entered that part...
3. People coming from searchengines will automatically land on the sid-less forum. What do you do with them?
4. How do you want to pass the information between pageviews at "which forum" somebody is looking? -> Add yet another get-variable to the URL.

Although I am not saying it can't be done, that is not my choice/way to do it.
If I get spidered, I'll just add the user-agent to the my sessions.php and be done with it. A lot easier and hardly any confusion for users.:)



Good questions.

TO get around all of those obstacles what I envision is simply a "spider page". To force ANY search engine to index all the threads.

The page would be linked to in the overall header invisibly, this page would run a script similar to a "latest posts" mod. It would simply list ALL the threads with links that do not have session ID's .

When threads are linked to in this way search engines don't seem to have a problem indexing them. This workaround would expand on your mod so that people who do thourough search engine manipulating will be able to target other search engines.

Do you think this idea would work?
User avatar
ilpil
Registered User
Posts: 98
Joined: Fri Dec 21, 2001 9:20 pm

Post by ilpil »

Is Lycos using spiders aswell? In the same way as google?
So could I just add

Code: Select all

&& !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Lycosbot') &&
Dont know if it's called Lycosbot though :)
radmanics
Registered User
Posts: 307
Joined: Sat Mar 30, 2002 3:29 pm
Location: England
Contact:

Post by radmanics »

this seems to be going in all directions now. I think someone should try and find a list of every bot/user agent that the major search engines use. Then it is a simple alteriseation to R.U.Seroius' MOD to make it work. Hopefully this would then be included in the next version of phpBB. ^^
User avatar
cdkrg
Registered User
Posts: 706
Joined: Fri Jul 12, 2002 12:35 pm
Contact:

Post by cdkrg »

The problem would be the undeclared spiders.

Google has a few and since they only come around every few months I'd not like to miss any.

But if it's easier we can work together to make a comprehensive list, I just don't know how we would alert everyone to update the mod if spiders change names.
jk1
Registered User
Posts: 103
Joined: Sun Jul 21, 2002 10:55 pm
Location: USA

Post by jk1 »

While searching around I came across this page: http://www.robotstxt.org/wc/active.html. Perhaps it might be helpful in making a list.

Also, I happen to be running awstats and it recorded 403 hits in 1 day from Road Runner: The ImageScape Robot, so I guess that one doesn't mind SIDs.
R. U. Serious
Registered User
Posts: 830
Joined: Mon Feb 11, 2002 2:07 pm

Post by R. U. Serious »

Just a quick feedback: Googlbot is just working at my site right now. It's working its way slowly through all the forum's links. So it seems this working out fine. :)

Thanx again Showscout for pointing it out :!:
pfilias
Registered User
Posts: 51
Joined: Tue Sep 04, 2001 4:52 pm

Post by pfilias »

Would this google mod also index private forums?
netclectic
Former Team Member
Posts: 4439
Joined: Wed Mar 13, 2002 3:08 pm
Location: Omnipresent
Contact:

Post by netclectic »

I would hope not, cos it caches a copy and makes it publicly available! 8O
Defend the game:
Image
pfilias
Registered User
Posts: 51
Joined: Tue Sep 04, 2001 4:52 pm

Post by pfilias »

Right, so any answers?
fishfreek
Registered User
Posts: 695
Joined: Tue May 14, 2002 3:05 pm
Location: Virginia
Contact:

Post by fishfreek »

I would think private forums would not get indexed since the google bot basicly is a visitor to the site so it would have access to all the pages and forums that a guest would have access to. Anything that requires a login to have access to should not be available to a search bot.
Have you joined the PetAdvice Network?

Are you in or around the Shenandoah Valley area? If so ask me how to join our local Shenandoah Valley aquarium group.
pfilias
Registered User
Posts: 51
Joined: Tue Sep 04, 2001 4:52 pm

Post by pfilias »

Ahhh... cool. Most of my forums require that you are registered to view them anyway... hmmmm.....
User avatar
ilpil
Registered User
Posts: 98
Joined: Fri Dec 21, 2001 9:20 pm

Post by ilpil »

Is there a way to disable that Googlebots show up as Guests in the Who is Online and Viewonline-files. Because they aren't really guests :) So, basically I want them NOT to show up on the lists at all. Not on the "most users ever online" either.

Can this be done?

Thanks!
netclectic
Former Team Member
Posts: 4439
Joined: Wed Mar 13, 2002 3:08 pm
Location: Omnipresent
Contact:

Post by netclectic »

I was wondering about this myself as it seems the googlebot has more than doubled the record number of users online for my forums which looks good but just isn't right :(


Anyway i came up with this, it seems simple enough and i cant see any reason that it would maybe have any undesired effects...

in includes/page_header.php
find

Code: Select all

if ( $row['session_ip'] != $prev_session_ip )
replace with

Code: Select all

		if ( ($row['session_ip'] != $prev_session_ip) && (!strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Googlebot')) )
Defend the game:
Image
netclectic
Former Team Member
Posts: 4439
Joined: Wed Mar 13, 2002 3:08 pm
Location: Omnipresent
Contact:

Post by netclectic »

[Edit]Doh!!!

Worked out why it wasn't working :roll:
I was comparing the user agent of the current user instead of the user agent of the user from the sessions table (which isn't stored anyway)!

Please excuse my blonde moment! :?
netclectic wrote: Ok, for some reason that doesn't seem to work as expected :?

[cut]...[/cut]

Anybody any ideas why i can record the user agent details as
but the code

Code: Select all

stristr($HTTP_USER_AGENT ,'Googlebot')
doesnt seem to return a match?


[/Edit]
Last edited by netclectic on Sun Sep 01, 2002 9:25 pm, edited 1 time in total.
Defend the game:
Image
R. U. Serious
Registered User
Posts: 830
Joined: Mon Feb 11, 2002 2:07 pm

Post by R. U. Serious »

Because you are running a newwer PHP version and register globals is off?

Try $HTTP_SERVER_VARS['HTTP_USER_AGENT'] instead of just $HTTP_USER_AGENT. That should work.
Locked

Return to “[2.0.x] MODs in Development”