Solve this SEO Issue Once and For All

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 2801
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman » Thu Jul 12, 2018 12:31 pm

AmigoJack wrote:
Thu Jul 12, 2018 12:15 pm
Frank Rizzo wrote:
Thu Jul 12, 2018 11:08 am
why do phpbb offer it
Times and search engines change, and phpBB is not that good in recognizing all bots either. I guess the main intent was to see bots at all when crawling a board, instead of ignoring them.
It's useful in different ways, for example the online list is more accurate. One way I have used it is to create additional adsbot group. This allows me to grant access to Googles adbot to bot restricted forums.

User avatar
Mick
Support Team Member
Support Team Member
Posts: 20237
Joined: Fri Aug 29, 2008 9:49 am
Location: Cardiff

Re: Solve this SEO Issue Once and For All

Post by Mick » Thu Jul 12, 2018 2:16 pm

So, you’re allowing certain bots to crawl where guests and some registered users have no access?
"The more connected we get the more alone we become" - Kyle Broflovski

There are no ‘threads’ in phpBB, they are topics.

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 2801
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman » Thu Jul 12, 2018 2:56 pm

Mick wrote:
Thu Jul 12, 2018 2:16 pm
So, you’re allowing certain bots to crawl where guests and some registered users have no access?
No, I have a few forums that are blocked to bots through robots.txt and phpBB permissions. All are available to public with the exception of one that is available to registered users. I think I know where you are going with this and I'm not concerned with someone spoofing the user agent to get access since they don't need to do that to begin with.

I'm displaying ads on those forums so Googles Mediapartners-Google needs access to it. Otherwise it will serve lousy ads or maybe none at all. in robots.txt at the top you put this:

Code: Select all

User-agent: Mediapartners-Google
Allow: /
This bot is also given access to those forums through phpBB permissions denied to all other bots. Just so it's clear you can display ads behind a login, as long as they are content pages is really the only thing that matters. You can even provide this bot with login credentials in your adsense account if you needed to. You don't need to do that with phpBB because you can identify it. ;)

User avatar
A_Jelly_Doughnut
Former Team Member
Posts: 34452
Joined: Sat Jan 18, 2003 1:26 am
Location: Where the Rivers Run
Contact:

Re: Solve this SEO Issue Once and For All

Post by A_Jelly_Doughnut » Tue Jul 17, 2018 4:00 pm

Frank Rizzo wrote:
Thu Jul 12, 2018 11:08 am
Hmm. If serving bots a different 'view' to regular users is such an issue why do phpbb offer it as a configuration option?
Back when this feature was added, some 14 years ago, bots were believed to be unable to handle the SIDs. Therefore, the main feature as designed is to forcibly remove the SID parameter for the bot user group.

Also, vBulletin at the time had a feature which served a somewhat cut-down style (without avatars, for example) to bots to cut down on bandwidth consumption. Way back then, Google didn't crawl all that frequently. Webmasters would often notice a spike in traffic from Googlebot on one particular day about once a month, a phenomenon that became known as the Googledance, which could tax servers with the extra traffic - thus the desire for a way to cut that down. By default, phpBB serves the same page to both bots and normal users though.
A Donut's Blog
"Bach's Prelude (Cello Suite No. 1) is driving Indiana country roads in Autumn" - Ann Kish

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: pikachuturkey, rautamiekka and 46 guests