[INFO] How gogle PHPBB!

[cdkrg]

1. How are you going to "hide" the links? -> You really can't, only make them less obvious.
2a. Do you want to duplicate each link? -> confusion & pagesize
2b. Only duplicate one link on the index? -> every spidered page will be on level deeper and possibly have a worse rank. Still confusion if normal users entered that part...
3. People coming from searchengines will automatically land on the sid-less forum. What do you do with them?
4. How do you want to pass the information between pageviews at "which forum" somebody is looking? -> Add yet another get-variable to the URL.

Although I am not saying it can't be done, that is not my choice/way to do it.
If I get spidered, I'll just add the user-agent to the my sessions.php and be done with it. A lot easier and hardly any confusion for users.

Good questions.

TO get around all of those obstacles what I envision is simply a "spider page". To force ANY search engine to index all the threads.

The page would be linked to in the overall header invisibly, this page would run a script similar to a "latest posts" mod. It would simply list ALL the threads with links that do not have session ID's .

When threads are linked to in this way search engines don't seem to have a problem indexing them. This workaround would expand on your mod so that people who do thourough search engine manipulating will be able to target other search engines.

Do you think this idea would work?

[ilpil]

Is Lycos using spiders aswell? In the same way as google?
So could I just add

Code: Select all

&& !strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Lycosbot') &&

Dont know if it's called Lycosbot though

[radmanics]

this seems to be going in all directions now. I think someone should try and find a list of every bot/user agent that the major search engines use. Then it is a simple alteriseation to R.U.Seroius' MOD to make it work. Hopefully this would then be included in the next version of phpBB. ^^

[cdkrg]

The problem would be the undeclared spiders.

Google has a few and since they only come around every few months I'd not like to miss any.

But if it's easier we can work together to make a comprehensive list, I just don't know how we would alert everyone to update the mod if spiders change names.

[jk1]

While searching around I came across this page: http://www.robotstxt.org/wc/active.html. Perhaps it might be helpful in making a list.

Also, I happen to be running awstats and it recorded 403 hits in 1 day from Road Runner: The ImageScape Robot, so I guess that one doesn't mind SIDs.

[R. U. Serious]

Just a quick feedback: Googlbot is just working at my site right now. It's working its way slowly through all the forum's links. So it seems this working out fine.

Thanx again Showscout for pointing it out

[pfilias]

Would this google mod also index private forums?

[netclectic]

I would hope not, cos it caches a copy and makes it publicly available! 8O

[pfilias]

Right, so any answers?

[fishfreek]

I would think private forums would not get indexed since the google bot basicly is a visitor to the site so it would have access to all the pages and forums that a guest would have access to. Anything that requires a login to have access to should not be available to a search bot.

[pfilias]

Ahhh... cool. Most of my forums require that you are registered to view them anyway... hmmmm.....

[ilpil]

Is there a way to disable that Googlebots show up as Guests in the Who is Online and Viewonline-files. Because they aren't really guests So, basically I want them NOT to show up on the lists at all. Not on the "most users ever online" either.

Can this be done?

Thanks!

[netclectic]

I was wondering about this myself as it seems the googlebot has more than doubled the record number of users online for my forums which looks good but just isn't right


Anyway i came up with this, it seems simple enough and i cant see any reason that it would maybe have any undesired effects...

in includes/page_header.php
find

Code: Select all

if ( $row['session_ip'] != $prev_session_ip )

replace with

Code: Select all

		if ( ($row['session_ip'] != $prev_session_ip) && (!strstr($HTTP_SERVER_VARS['HTTP_USER_AGENT'] ,'Googlebot')) )

[netclectic]

[Edit]Doh!!!

Worked out why it wasn't working
I was comparing the user agent of the current user instead of the user agent of the user from the sessions table (which isn't stored anyway)!

Please excuse my blonde moment!

Ok, for some reason that doesn't seem to work as expected

[cut]...[/cut]

Anybody any ideas why i can record the user agent details as

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

but the code

Code: Select all

stristr($HTTP_USER_AGENT ,'Googlebot')

doesnt seem to return a match?

[/Edit]

[R. U. Serious]

Because you are running a newwer PHP version and register globals is off?

Try $HTTP_SERVER_VARS['HTTP_USER_AGENT'] instead of just $HTTP_USER_AGENT. That should work.