Solve this SEO Issue Once and For All

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 10:44 am

11 years ago I posted a thread stating Out of the box phpBB 3 is awful for SEO. 11 years on and I still hold that view.

tl;dr
15 year old forum is crawled by google but thinly indexed. Site is hardly crawled by bing. Searching for feature rich topics never comes up in SERPS, which must be losing me a lot of traffic.

Preamble
I have run a forum for 15 years, on a site, which has been going for 18 years. Overall 222,442 posts have been made (some deleted) from 18,361 topics (some deleted). The whole site has 22,500* pages showing in google serps, and about 305,000 pages showing in bing serps.
*google also used to have 300,000 indexed but some of the pages are name listings which google has optimised

Issue
Whilst the main site has figured well in serps over the years, the form has never figured well. Searching for key rich topics and phrases is fruitless. Rather than dozens of pages from the forum showing up in search only pages from the main site are returned. If you be more specific by putting quotes around a phrase you may get just 1 result returned (there should be more).

Google is crawling the site - but is hardly returning results. site:www.flatstats.co.uk/ppp/ shows there to be 2,240 results from the forum but rarely is a forum post returned in serps.

Bing is hardly crawling the site at all. site:www.flatstats.co.uk/ppp/ shows there to be just 11 results

What Have I Done?
I have tried everything that was topical at the time (and this may be a problem in the later years). Right at the start I did all the recommendations for phpbb2 (such as removing sid for bots, blocking some resources in robots, which could cause duplicates) For a few years after I tried the recommendations for phpbb3. But I have hardly changed anything since 3.2 came out.

What Could Be The Main Issue?
* For google maybe the lack of meta information?

* Google WMT does show that one of the 11 sitemaps does have 3,168 warnings:

"Sitemap contains urls which are blocked by robots.txt."

That must be due to the anti-duplicates blocking in robots.txt.

* For bing I do not have a clue. All I know is that bing is a very heavy crawler of the whole site but just won't go near the forum. It's properly registered as a bot and has bot access to the forums. There is no block for it in robots.txt

What Now?
Do I need to follow the 2017/2018 recommendations and wait a couple more years? :D Or is there something fundamentally wrong with my forum setup, which has been in there right from the start?

What do I need to do to finally get SEs to return pages from my forum?

Bots Info
bots.jpg

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 11:19 am

Bing
Looking at Bing Webmaster admin panel I observe the following:

Configure My Site -> Ignore URL Parameters
f
t
p

are listed. Could this cause a problem? In Google WMT f, t, and p are listed with 'Let Googlebot decide'. If they are listed in Bing webmaster admin panel does this mean any url with those parameters is ignored?

Configure My Site -> Crawl Control
I have this turned down to the minimum one block for all 24 hours because bing does crawl the site too frequently - much more frequently than google. Google crawls around 2-3000 a day, bing 15-20,000.

Could it be that bing is so fixated on crawling the ~280,000 main site pages that it will not crawl the forum pages?

User avatar
Ger
Recognised Extension Developer
Posts: 1791
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Solve this SEO Issue Once and For All

Post by Ger » Wed Jul 11, 2018 11:25 am

Frank Rizzo wrote:
Wed Jul 11, 2018 10:44 am
What do I need to do to finally get SEs to return pages from my forum?
Remove all so-called "optimisations" .

Without any changes to robots, without any sitemap and with only minor changes to the style, my good old forum is indexed very well. 1970 results while there are 2071 topics. We have some hidden forums so I think every public readable topic is properly indexed.

It's a very small, localised niche board. So I'd say your statement from 11 years ago is false.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 11:33 am

Yep, my forum is indexed in google too. It's just not coming up in serps when searching for terms.

User avatar
Ger
Recognised Extension Developer
Posts: 1791
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Solve this SEO Issue Once and For All

Post by Ger » Wed Jul 11, 2018 11:41 am

Frank Rizzo wrote:
Wed Jul 11, 2018 11:33 am
Yep, my forum is indexed in google too. It's just not coming up in serps when searching for terms.
I'm not sure what you mean here. Do you mean that when someone searches for "horse racing" you're not on the top?

That's probably because there are thousands of competitors on that topic.

Also; it seems your forum isn't indexed that well. about 2210 topics indexed while you have over 18,000 topics.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 11:49 am

Looks like it's 2007 again :x

User avatar
Ger
Recognised Extension Developer
Posts: 1791
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Solve this SEO Issue Once and For All

Post by Ger » Wed Jul 11, 2018 12:01 pm

Frank Rizzo wrote:
Wed Jul 11, 2018 11:49 am
Looks like it's 2007 again :x
Not sure what you mean by this.
It might help if you actually respond to my question where I try to understand you and try to help you see where the actual problem lies.

If instead you intend to have a place to rant a bit, than yeah, this might lead to a locked topic just like in 2007.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
Mick
Support Team Member
Support Team Member
Posts: 20338
Joined: Fri Aug 29, 2008 9:49 am
Location: Cardiff

Re: Solve this SEO Issue Once and For All

Post by Mick » Wed Jul 11, 2018 12:04 pm

The thing is, your last topic (not thread, there are none in phpBB) was quite extensive, fifteen pages extensive to be precise. Since then there have been a few topics on the subject but no-one has stood up and said "look, I've got a 100% working, SEO kit for phpBB", why? Well I'd guess nobody is that interested apart from the few SEO stalwarts and, as has been said a thousand and one times, phpBB does ok without any extra bells and whistles. You may not believe that to be true but I believe it.
"The more connected we get the more alone we become" - Kyle Broflovski

There are no ‘threads’ in phpBB, they are topics.

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 2:53 pm

Ger wrote:
Wed Jul 11, 2018 12:01 pm
Frank Rizzo wrote:
Wed Jul 11, 2018 11:49 am
Looks like it's 2007 again :x
Not sure what you mean by this.
It might help if you actually respond to my question where I try to understand you and try to help you see where the actual problem lies.

If instead you intend to have a place to rant a bit, than yeah, this might lead to a locked topic just like in 2007.
Well when you say things like

"That's probably because there are thousands of competitors on that topic. "

you make it sound as if I am a noob who has just set up a wix site. Come on; a bit of respect for us veterans here :D

Optimisations were removed when 3.2 was installed a while ago. The only 'tinkering' that is now in use is in the robots.txt. I believe most of these are totally essential to stop duplication penalties and to stop leaking of info you don't want leaked.

This is now what I have in the file:

Code: Select all

Disallow: /ppp/adm/
Disallow: /ppp/cache/
Disallow: /ppp/config/
Disallow: /ppp/develop/
Disallow: /ppp/docs/
Disallow: /ppp/files/
Disallow: /ppp/images/
Disallow: /ppp/includes/
Disallow: /ppp/install/
Disallow: /ppp/language/
Disallow: /ppp/store/
Disallow: /ppp/styles/

Disallow: /ppp/common.php
Disallow: /ppp/config.php
Disallow: /ppp/cron.php
Disallow: /ppp/download.php
Disallow: /ppp/faq.php
Disallow: /ppp/index.html
Disallow: /ppp/login.php
Disallow: /ppp/mcp.php
Disallow: /ppp/memberlist.php
Disallow: /ppp/posting.php
Disallow: /ppp/profile.php
Disallow: /ppp/report.php
Disallow: /ppp/search.php
Disallow: /ppp/style.php
Disallow: /ppp/ucp.php
Disallow: /ppp/viewonline.php

Disallow: /ppp/viewforum.php?f=1
Disallow: /ppp/*.php?f=1&*
The last two lines were added due to a recent-ish post by The Coalman.

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 3:00 pm

Mick wrote:
Wed Jul 11, 2018 12:04 pm
The thing is, your last topic (not thread, there are none in phpBB) was quite extensive, fifteen pages extensive to be precise. Since then there have been a few topics on the subject but no-one has stood up and said "look, I've got a 100% working, SEO kit for phpBB", why? Well I'd guess nobody is that interested apart from the few SEO stalwarts and, as has been said a thousand and one times, phpBB does ok without any extra bells and whistles. You may not believe that to be true but I believe it.
Yep and just to make things clear the 2007 thread, sorry topic, was not closed due to abuse. It was closed due to some of the techniques being no longer relevant. I have started this 2018 as advised to by the moderator.

Let's not go over old ground with "It does work", "it works for me", "it works for phpbb". Let's try and solve a few simple issues with my specific site (of which I am sure thousands of boards have exactly the same issue).

Question 1. Why is google crawling my *forum* when bing does not?
Just to expand on that for clarity: bing crawls the whole site, google crawls the whole site, bing does not crawl the forum, google does crawl the forum. Why?

User avatar
Lumpy Burgertushie
Registered User
Posts: 65301
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Solve this SEO Issue Once and For All

Post by Lumpy Burgertushie » Wed Jul 11, 2018 3:04 pm

I think part of the problem is that what worked and was needed 10 years ago may not apply. since then, google has learned a lot about how dynamic websites like phpbb work. there is no longer a problem with the duplicate penalties for dynamic sites.

there is no longer a need for removing the sid, there is no reall need to even use a robots.txt file for phpbb.

I would get rid of all of that and start with a clean slate.

if you really want to help phpbb then yes, meta tags for every forum/topic/post will certainly help.
there is an extension I believe that will create that for you.
'
I haven't looked at your board but if your forum titles and topic titles are actually descriptive and if they contain some of those keywords that are relevant to the board subject etc. , that will help a lot as well.

your assertion that phpbb is terrible for SEO is simply not the case for many thousands of standard "out of the box" phpbb installations around the world.

as for bing not crawling, I suggest it is just what you said. you have told it not to crawl anything with f or p in the filename. so, it does not worry about any forum page or post etc.


robert
I am available for custom work on a donation basis. Please send me a PM with your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

OK, so what's the speed of dark?

User avatar
Ger
Recognised Extension Developer
Posts: 1791
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Solve this SEO Issue Once and For All

Post by Ger » Wed Jul 11, 2018 3:05 pm

Frank Rizzo wrote:
Wed Jul 11, 2018 2:53 pm
Optimisations were removed when 3.2 was installed a while ago. The only 'tinkering' that is now in use is in the robots.txt. I believe most of these are totally essential to stop duplication penalties and to stop leaking of info you don't want leaked.
that doesn't help at all.

Robots.txt is not intended to prevent leaking of information since no bot needs to respect what's in it.

Also, those lines do nothing about duplicates. IMO, you should just remove those lines from robots.txt. It does more harm than good.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 2806
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman » Wed Jul 11, 2018 3:19 pm

Frank Rizzo wrote:
Wed Jul 11, 2018 2:53 pm
The last two lines were added due to a recent-ish post by The Coalman.
My apologies but this....

Code: Select all

Disallow: /ppp/viewforum.php?f=1
will block any forum starting with a one. e.g. /ppp/viewforum.php?f=123

It needs to be:

Code: Select all

Disallow: /ppp/viewforum.php?f=1$
You should have Google Webmaster account, you can always see what is being blocked and test these things.

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 2806
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Solve this SEO Issue Once and For All

Post by thecoalman » Wed Jul 11, 2018 3:29 pm

Ger wrote:
Wed Jul 11, 2018 3:05 pm
that doesn't help at all.

Robots.txt is not intended to prevent leaking of information since no bot needs to respect what's in it.

Also, those lines do nothing about duplicates. IMO, you should just remove those lines from robots.txt. It does more harm than good.
Those lines would prevent a bot from making requests to forums and the topics in them you do not want indexed. This prevents unnecessary requests to your server and also frees the bot resources to index or reindex other content on your site you do want indexed.

Frank Rizzo
Registered User
Posts: 135
Joined: Sun Jan 05, 2003 11:10 pm
Contact:

Re: Solve this SEO Issue Once and For All

Post by Frank Rizzo » Wed Jul 11, 2018 3:41 pm

Lumpy Burgertushie wrote:
Wed Jul 11, 2018 3:04 pm
as for bing not crawling, I suggest it is just what you said. you have told it not to crawl anything with f or p in the filename. so, it does not worry about any forum page or post etc.
robert
All points noted. Thank you.

Specifically for bing those f, p, t I clearly got that one. My assumption was that bing would ignore just the parameters (similar to google) but in this case it must be totally ignoring any url that has an f, p, or t parameter. That's hell of a gotcha. I have removed these and will confirn if that fixes the crawl in a few days.

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: No registered users and 24 guests