Bot Question

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
War Horse
Registered User
Posts: 67
Joined: Sat Oct 04, 2008 6:53 pm

Bot Question

Post by War Horse » Fri Aug 21, 2009 5:46 pm

With private forums (ones that require registration for users to see forums) is there any harm in having the googlebot (or bots in general) spider the site? If there is harm, is the best way to dissallow them access by "deactivating" them in the control panel?

User avatar
Dog Cow
Registered User
Posts: 2494
Joined: Fri Jan 28, 2005 12:14 am
Contact:

Re: Bot Question

Post by Dog Cow » Fri Aug 21, 2009 5:49 pm

Do you think that Google bot will register for your site and log in to read the forums?
Moof!
Mac GUI Vault: Retro Apple II & Macintosh computing archive.
Inside Allerton bookMac GUIMac 512K Blog

War Horse
Registered User
Posts: 67
Joined: Sat Oct 04, 2008 6:53 pm

Re: Bot Question

Post by War Horse » Fri Aug 21, 2009 5:52 pm

Dog Cow wrote:Do you think that Google bot will register for your site and log in to read the forums?
:lol: No. Just don't see the reason to have the data and images within indexed anywhere.

User avatar
Dog Cow
Registered User
Posts: 2494
Joined: Fri Jan 28, 2005 12:14 am
Contact:

Re: Bot Question

Post by Dog Cow » Fri Aug 21, 2009 5:53 pm

Make a robots.txt file and disallow the Googlebot user agent, and any other search robots if you want.

The phpBB control panel option won't stop them from coming.
Moof!
Mac GUI Vault: Retro Apple II & Macintosh computing archive.
Inside Allerton bookMac GUIMac 512K Blog

User avatar
Brf
Support Team Member
Support Team Member
Posts: 51868
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: Bot Question

Post by Brf » Fri Aug 21, 2009 5:55 pm

If you disable the bots, they will be treated as unregisitered guests, but there is no need to do that. Just remove the Forum Permissions for the Bots usergroup, and they will not see the forums.

User avatar
mals69
Registered User
Posts: 595
Joined: Thu Mar 30, 2006 6:37 am

Re: Bot Question

Post by mals69 » Tue Aug 25, 2009 7:03 am

To my knowledge Google will not spider any website material that is not open to the public, private forums are not spidered. Google wants their search results publicly available. :P

User avatar
Brf
Support Team Member
Support Team Member
Posts: 51868
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: Bot Question

Post by Brf » Tue Aug 25, 2009 10:52 am

That is not true.
As I said, the Bots group is separate from the Guests group. Therefore, bots can be given permissions to read forums that guests cannot and vice-versa

User avatar
mals69
Registered User
Posts: 595
Joined: Thu Mar 30, 2006 6:37 am

Re: Bot Question

Post by mals69 » Tue Aug 25, 2009 11:21 am

Why would Google bothering spidering material that is not for public viewing that people have to take the time to register for, not be much of a search engine. Like I said to my knowledge, 3 1/2 years of Google bots and never spidered our private off topic forum but spiders the other 13 topics that are for public viewing.

Robot text files another thing all together, yes you can instruct them to go here and not go there on your site, but non public material Google makes up its own mind not to spider in my experience.

If other sites have private forums that somehow get spidered the private material will not be indexed in Google search results, so the topic starter has nothing to worry about in the first place unless on a tiny hosting plan and has to worry about bandwidth. :P

User avatar
drathbun
Former Team Member
Posts: 12204
Joined: Thu Jun 06, 2002 3:51 pm
Location: TOPICS_TABLE
Contact:

Re: Bot Question

Post by drathbun » Tue Aug 25, 2009 1:39 pm

More importantly, why would any board owner give a bot permission to read something private? Then all someone has to do is use google to read the private forums; doesn't make sense.
I blog about phpBB: phpBBDoctor blog
Still using phpbb2? So am I! Click below for details
Image

Roberdin
Registered User
Posts: 203
Joined: Sun Nov 24, 2002 3:56 pm
Location: London, United Kingdom

Re: Bot Question

Post by Roberdin » Tue Aug 25, 2009 2:11 pm

mals69 wrote:Why would Google bothering spidering material that is not for public viewing that people have to take the time to register for, not be much of a search engine. Like I said to my knowledge, 3 1/2 years of Google bots and never spidered our private off topic forum but spiders the other 13 topics that are for public viewing.
That's not how it works. When a bot (a member of the Bots group) visits a phpBB forum, phpBB recognises the "bot" as such, and grants the bot appropriate permissions. These permissions could be different to those of a "Guest", if you decided to configure it that way. In this case, the bot could spider topics that a guest could not see.
Roberdin

User avatar
mals69
Registered User
Posts: 595
Joined: Thu Mar 30, 2006 6:37 am

Re: Bot Question

Post by mals69 » Tue Aug 25, 2009 8:57 pm

I agree with drathbun, like the last paragraph in my last post, if peoples private forums are being spidered it does not matter cause the spidering results will not show in Googles search index, so why have others in here bothered giving instructions on robot txt instructions for a private forum ? If some want to create un-necessary work for themselves be my guest. :) 8-)

User avatar
mals69
Registered User
Posts: 595
Joined: Thu Mar 30, 2006 6:37 am

Re: Bot Question

Post by mals69 » Wed Aug 26, 2009 3:56 am

Got it wrong on Google not spidering private forums - sorry folks, :oops:

Googles system surely smart enough to know not to keep spidering a private forum ? It knows not to show any private forums posted material in their search results so why are private forums continually spidered and more importantly what is happening with the material from these private forums ? :?

User avatar
onehundredandtwo
Registered User
Posts: 1228
Joined: Fri Nov 14, 2008 8:07 am

Re: Bot Question

Post by onehundredandtwo » Wed Aug 26, 2009 6:20 am

To check if it's still private. ;)

I've noticed Googlebot will always index every part of one of my sites, it doesn't matter if it gets a phpBB error or not.
Need help preventing spam? Read Preventing spam in phpBB 3.0.6 and above

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Bot Question

Post by Eelke » Wed Aug 26, 2009 6:23 am

All Google does is load web pages and follow links on them. They don't have any magic way to get into any sites and they are not being "smart" about indexing anything; basically they index anything they can get to. In most cases, they would not see anything different than a regular web user that does not log into sites.

With that said, phpBB recognizes friendly bots like Google and allows you to assign them different permissions then regular guests, by assigning permissions to the bots group. That way, it's possible to allow Google to get into places where guests can not go. Is that smart? Probably not. The specific bots permissions are best used to only shield off things that really make no sense to spider, eventhough you want guest users to be able to see them (which is the other way around; bot permissions are usually a bit more restrictive than guest permissions, not less).

User avatar
Brf
Support Team Member
Support Team Member
Posts: 51868
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: Bot Question

Post by Brf » Wed Aug 26, 2009 11:32 am

mals69 wrote:Googles system surely smart enough to know not to keep spidering a private forum ? It knows not to show any private forums posted material in their search results
As Eelke is trying to explain, Google and other bots do not "know" anything. All they do is follow links.
If a usergroup is not given permissions on a forum, then there will be no link for follow. If you give the Bots usergroup permissions on a private forum, the link will show and Googlebot will follow and index that private forum. Those indexed links could probably not be followed by a guest, but the private pages would still be indexed and cached.

As for drathbun's question on motivation to do this... A searching guest might find an interesting subject, but not be able to see it except in cache. That guest might be motivated to register to participate in that subject, or at least read it in a non-cached page.

Post Reply

Return to “phpBB Discussion”