Error HTTP 403 - Google fails to index forum pages

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

GoogleBot is directed to forum pages via sitemap and crawls them, but refuses to index them, returning HTTP error 403. Forum pages display OK in browser.

Checked for server issues with hosting company. Checked robots.txt file. Checked htaccess file.

Tried Fetch as Google and forum pages return error.

All other pages on web site are crawled and indexed OK.

Anyone got any other ideas as to what may be the problem, or how I can further diagnose ?

Thanks in advance...
User avatar
david63
Registered User
Posts: 18835
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by david63 »

Washerman wrote:
Mon Feb 19, 2018 3:58 pm
GoogleBot is directed to forum pages via sitemap
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored
User avatar
Lumpy Burgertushie
Registered User
Posts: 68471
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by Lumpy Burgertushie »

403 error means you do not have permission to view this page.

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 51807
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by stevemaury »

And Sitemaps do not make much sense for dynamic sites, like phpBB.
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

Anyone got any idea as to what may be causing the problem, or how I can further diagnose the cause of this problem ?
User avatar
Mick
Support Team Member
Support Team Member
Posts: 22951
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket probably.

Re: Error HTTP 403 - Google fails to index forum pages

Post by Mick »

david63 wrote:
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
"The more connected we get the more alone we become" - Kyle Broflovski©
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

Mick, the sitemap is not the cause of the problem, but thanks for your input
User avatar
Mick
Support Team Member
Support Team Member
Posts: 22951
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket probably.

Re: Error HTTP 403 - Google fails to index forum pages

Post by Mick »

May be but phpBB does not have a sitemap so how are we to decide what the issue is? How do you know the issue isn’t with the sitemap? If you leave phpBB to it’s own devices you don’t need one anyway.
"The more connected we get the more alone we become" - Kyle Broflovski©
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

Mick, there are 19 pages in the sitemap. On submitting it to Google Search Console, all 19 pages are detected and accepted. 15 of the pages (non forum pages) are all indexed, but the balance (4 pages) will not index - they return an error 403 - even though they display in the browser. I have checked all the things that I mentioned in the initial post, what else could be causing the pages to return an error 403 ?
User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 51807
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by stevemaury »

Mick wrote:
Mon Feb 19, 2018 8:32 pm
david63 wrote:
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)
User avatar
Lumpy Burgertushie
Registered User
Posts: 68471
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by Lumpy Burgertushie »

and, the 403 error means that the google bot does not have permsission to view those 4 pages.

in phpbb the bots group has certain permissions set. check your bot group permissions and /or your guest permissions to see if they have correct permissions set to view those pages.


robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4259
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by thecoalman »

As Lumpy suggested the bot group is likely denied access to those pages. The easiest way to test how a bot sees your site is most browsers have an extension to switch the user agent to Googlebot for example.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

Hi thecoalman, thank you for your help and advice. I already used the tool @ https://httpstatus.io/ with the request headers set to GoogleBot and the response in the html source code is "You are not authorised to read this forum."

Lumpy Burgertushie, thank you for your input. I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-

Post = All set to "No"
Content = All set to "No"
Actions = All set to "No" except "Can Download Files ", "Can see Forum", "Can Print Topics" and "Can Read Forum"
Polls = All set to "No"

Are there any other permissions that I am overlooking, or is there anything else that I should consider looking at ?
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4259
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by thecoalman »

Washerman wrote:
Tue Feb 20, 2018 6:17 am
I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-
This only set default permissions when permissions are applied.*

ACP >> Permisssions Tab >> Forum Permissions >> Select the Forum(s) you want to apply permissions to

One the right hand side is all the groups that have permissions set for the selected forums, the bot group is likely in the bottom box. You will need to highlight it and hit the button "add permissions". Slightly off topic but the easiest way to deny access to group is to never add them to the forum. For example if you had private forum for admins and moderators you would only add those groups. All other groups would be denied access by default.

Once you have added or removed groups from the selected forums you can also highlight all the groups and select edit permissions. On this page you can set permissions, *using the role as one example.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman »

thecoalman... I knew that I must be overlooking something. Sincere thanks for your help and assistance - problem solved !
Post Reply

Return to “[3.2.x] Support Forum”