Error HTTP 403 - Google fails to index forum pages

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Mon Feb 19, 2018 3:58 pm

GoogleBot is directed to forum pages via sitemap and crawls them, but refuses to index them, returning HTTP error 403. Forum pages display OK in browser.

Checked for server issues with hosting company. Checked robots.txt file. Checked htaccess file.

Tried Fetch as Google and forum pages return error.

All other pages on web site are crawled and indexed OK.

Anyone got any other ideas as to what may be the problem, or how I can further diagnose ?

Thanks in advance...

User avatar
david63
Registered User
Posts: 16321
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by david63 » Mon Feb 19, 2018 4:25 pm

Washerman wrote:
Mon Feb 19, 2018 3:58 pm
GoogleBot is directed to forum pages via sitemap
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Lumpy Burgertushie
Registered User
Posts: 66479
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by Lumpy Burgertushie » Mon Feb 19, 2018 4:31 pm

403 error means you do not have permission to view this page.

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

If a tree falls in the forest and nobody is there, does it make a sound?

User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 50565
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by stevemaury » Mon Feb 19, 2018 5:12 pm

And Sitemaps do not make much sense for dynamic sites, like phpBB.
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)

Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Mon Feb 19, 2018 8:26 pm

Anyone got any idea as to what may be causing the problem, or how I can further diagnose the cause of this problem ?

User avatar
Mick
Support Team Member
Support Team Member
Posts: 21311
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket - definitely

Re: Error HTTP 403 - Google fails to index forum pages

Post by Mick » Mon Feb 19, 2018 8:32 pm

david63 wrote:
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
"The more connected we get the more alone we become" - Kyle Broflovski

Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Mon Feb 19, 2018 8:42 pm

Mick, the sitemap is not the cause of the problem, but thanks for your input

User avatar
Mick
Support Team Member
Support Team Member
Posts: 21311
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket - definitely

Re: Error HTTP 403 - Google fails to index forum pages

Post by Mick » Mon Feb 19, 2018 8:49 pm

May be but phpBB does not have a sitemap so how are we to decide what the issue is? How do you know the issue isn’t with the sitemap? If you leave phpBB to it’s own devices you don’t need one anyway.
"The more connected we get the more alone we become" - Kyle Broflovski

Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Mon Feb 19, 2018 9:09 pm

Mick, there are 19 pages in the sitemap. On submitting it to Google Search Console, all 19 pages are detected and accepted. 15 of the pages (non forum pages) are all indexed, but the balance (4 pages) will not index - they return an error 403 - even though they display in the browser. I have checked all the things that I mentioned in the initial post, what else could be causing the pages to return an error 403 ?

User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 50565
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by stevemaury » Mon Feb 19, 2018 10:18 pm

Mick wrote:
Mon Feb 19, 2018 8:32 pm
david63 wrote:
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)

User avatar
Lumpy Burgertushie
Registered User
Posts: 66479
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by Lumpy Burgertushie » Mon Feb 19, 2018 10:22 pm

and, the 403 error means that the google bot does not have permsission to view those 4 pages.

in phpbb the bots group has certain permissions set. check your bot group permissions and /or your guest permissions to see if they have correct permissions set to view those pages.


robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

If a tree falls in the forest and nobody is there, does it make a sound?

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 3257
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by thecoalman » Tue Feb 20, 2018 2:31 am

As Lumpy suggested the bot group is likely denied access to those pages. The easiest way to test how a bot sees your site is most browsers have an extension to switch the user agent to Googlebot for example.

Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Tue Feb 20, 2018 6:17 am

Hi thecoalman, thank you for your help and advice. I already used the tool @ https://httpstatus.io/ with the request headers set to GoogleBot and the response in the html source code is "You are not authorised to read this forum."

Lumpy Burgertushie, thank you for your input. I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-

Post = All set to "No"
Content = All set to "No"
Actions = All set to "No" except "Can Download Files ", "Can see Forum", "Can Print Topics" and "Can Read Forum"
Polls = All set to "No"

Are there any other permissions that I am overlooking, or is there anything else that I should consider looking at ?

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 3257
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Error HTTP 403 - Google fails to index forum pages

Post by thecoalman » Tue Feb 20, 2018 4:02 pm

Washerman wrote:
Tue Feb 20, 2018 6:17 am
I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-
This only set default permissions when permissions are applied.*

ACP >> Permisssions Tab >> Forum Permissions >> Select the Forum(s) you want to apply permissions to

One the right hand side is all the groups that have permissions set for the selected forums, the bot group is likely in the bottom box. You will need to highlight it and hit the button "add permissions". Slightly off topic but the easiest way to deny access to group is to never add them to the forum. For example if you had private forum for admins and moderators you would only add those groups. All other groups would be denied access by default.

Once you have added or removed groups from the selected forums you can also highlight all the groups and select edit permissions. On this page you can set permissions, *using the role as one example.

Washerman
Registered User
Posts: 9
Joined: Mon Feb 15, 2016 8:41 am

Re: Error HTTP 403 - Google fails to index forum pages

Post by Washerman » Tue Feb 20, 2018 5:24 pm

thecoalman... I knew that I must be overlooking something. Sincere thanks for your help and assistance - problem solved !

Post Reply

Return to “[3.2.x] Support Forum”