Solve this SEO Issue Once and For All

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5871
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman »

AmigoJack wrote: Thu Jul 12, 2018 12:15 pm
Frank Rizzo wrote: Thu Jul 12, 2018 11:08 amwhy do phpbb offer it
Times and search engines change, and phpBB is not that good in recognizing all bots either. I guess the main intent was to see bots at all when crawling a board, instead of ignoring them.
It's useful in different ways, for example the online list is more accurate. One way I have used it is to create additional adsbot group. This allows me to grant access to Googles adbot to bot restricted forums.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26502
Joined: Fri Aug 29, 2008 9:49 am

Re: Solve this SEO Issue Once and For All

Post by Mick »

So, you’re allowing certain bots to crawl where guests and some registered users have no access?
  • "The more connected we get the more alone we become" - Kyle Broflovski©
  • "The good news is hell is just the product of a morbid human imagination.
    The bad news is, whatever humans can imagine, they can usually create.
    " - Harmony Cobel
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5871
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman »

Mick wrote: Thu Jul 12, 2018 2:16 pm So, you’re allowing certain bots to crawl where guests and some registered users have no access?
No, I have a few forums that are blocked to bots through robots.txt and phpBB permissions. All are available to public with the exception of one that is available to registered users. I think I know where you are going with this and I'm not concerned with someone spoofing the user agent to get access since they don't need to do that to begin with.

I'm displaying ads on those forums so Googles Mediapartners-Google needs access to it. Otherwise it will serve lousy ads or maybe none at all. in robots.txt at the top you put this:

Code: Select all

User-agent: Mediapartners-Google
Allow: /
This bot is also given access to those forums through phpBB permissions denied to all other bots. Just so it's clear you can display ads behind a login, as long as they are content pages is really the only thing that matters. You can even provide this bot with login credentials in your adsense account if you needed to. You don't need to do that with phpBB because you can identify it. ;)
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
A_Jelly_Doughnut
Former Team Member
Posts: 34459
Joined: Sat Jan 18, 2003 1:26 am
Location: Where the Rivers Run
Contact:

Re: Solve this SEO Issue Once and For All

Post by A_Jelly_Doughnut »

Frank Rizzo wrote: Thu Jul 12, 2018 11:08 am Hmm. If serving bots a different 'view' to regular users is such an issue why do phpbb offer it as a configuration option?
Back when this feature was added, some 14 years ago, bots were believed to be unable to handle the SIDs. Therefore, the main feature as designed is to forcibly remove the SID parameter for the bot user group.

Also, vBulletin at the time had a feature which served a somewhat cut-down style (without avatars, for example) to bots to cut down on bandwidth consumption. Way back then, Google didn't crawl all that frequently. Webmasters would often notice a spike in traffic from Googlebot on one particular day about once a month, a phenomenon that became known as the Googledance, which could tax servers with the extra traffic - thus the desire for a way to cut that down. By default, phpBB serves the same page to both bots and normal users though.
A Donut's Blog
"Bach's Prelude (Cello Suite No. 1) is driving Indiana country roads in Autumn" - Ann Kish
Post Reply

Return to “phpBB Discussion”