Error HTTP 403 - Google fails to index forum pages

Washerman · Post by **Washerman** » Mon Feb 19, 2018 3:58 pm

GoogleBot is directed to forum pages via sitemap and crawls them, but refuses to index them, returning HTTP error 403. Forum pages display OK in browser.

Checked for server issues with hosting company. Checked robots.txt file. Checked htaccess file.

Tried Fetch as Google and forum pages return error.

All other pages on web site are crawled and indexed OK.

Anyone got any other ideas as to what may be the problem, or how I can further diagnose ?

Thanks in advance...

david63 · Post by **david63** » Mon Feb 19, 2018 4:25 pm

Washerman wrote: ↑
Mon Feb 19, 2018 3:58 pm
GoogleBot is directed to forum pages via sitemap

As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap

Lumpy Burgertushie · Post by **Lumpy Burgertushie** » Mon Feb 19, 2018 4:31 pm

403 error means you do not have permission to view this page.

robert

Post by **stevemaury** » Mon Feb 19, 2018 5:12 pm

And Sitemaps do not make much sense for dynamic sites, like phpBB.

Washerman · Post by **Washerman** » Mon Feb 19, 2018 8:26 pm

Anyone got any idea as to what may be causing the problem, or how I can further diagnose the cause of this problem ?

Post by **Mick** » Mon Feb 19, 2018 8:32 pm

david63 wrote: ↑
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap

Washerman · Post by **Washerman** » Mon Feb 19, 2018 8:42 pm

Mick, the sitemap is not the cause of the problem, but thanks for your input

Post by **Mick** » Mon Feb 19, 2018 8:49 pm

May be but phpBB does not have a sitemap so how are we to decide what the issue is? How do you know the issue isn’t with the sitemap? If you leave phpBB to it’s own devices you don’t need one anyway.

Washerman · Post by **Washerman** » Mon Feb 19, 2018 9:09 pm

Mick, there are 19 pages in the sitemap. On submitting it to Google Search Console, all 19 pages are detected and accepted. 15 of the pages (non forum pages) are all indexed, but the balance (4 pages) will not index - they return an error 403 - even though they display in the browser. I have checked all the things that I mentioned in the initial post, what else could be causing the pages to return an error 403 ?

Post by **stevemaury** » Mon Feb 19, 2018 10:18 pm

Mick wrote: ↑
Mon Feb 19, 2018 8:32 pm

david63 wrote: ↑
Mon Feb 19, 2018 4:25 pm
As there is no "sitemap" in phpBB I would suggest you ask your question at the place where you got your sitemap

Lumpy Burgertushie · Post by **Lumpy Burgertushie** » Mon Feb 19, 2018 10:22 pm

and, the 403 error means that the google bot does not have permsission to view those 4 pages.

in phpbb the bots group has certain permissions set. check your bot group permissions and /or your guest permissions to see if they have correct permissions set to view those pages.

robert

Post by **thecoalman** » Tue Feb 20, 2018 2:31 am

As Lumpy suggested the bot group is likely denied access to those pages. The easiest way to test how a bot sees your site is most browsers have an extension to switch the user agent to Googlebot for example.

Washerman · Post by **Washerman** » Tue Feb 20, 2018 6:17 am

Hi thecoalman, thank you for your help and advice. I already used the tool @ https://httpstatus.io/ with the request headers set to GoogleBot and the response in the html source code is "You are not authorised to read this forum."

Lumpy Burgertushie, thank you for your input. I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-

Post = All set to "No"
Content = All set to "No"
Actions = All set to "No" except "Can Download Files ", "Can see Forum", "Can Print Topics" and "Can Read Forum"
Polls = All set to "No"

Are there any other permissions that I am overlooking, or is there anything else that I should consider looking at ?

Post by **thecoalman** » Tue Feb 20, 2018 4:02 pm

Washerman wrote: ↑
Tue Feb 20, 2018 6:17 am
I have checked the forum permissions @ ACP>Permissions>Permission Roles>Forum Roles>Bot Access :-

This only set default permissions when permissions are applied.*

ACP >> Permisssions Tab >> Forum Permissions >> Select the Forum(s) you want to apply permissions to

One the right hand side is all the groups that have permissions set for the selected forums, the bot group is likely in the bottom box. You will need to highlight it and hit the button "add permissions". Slightly off topic but the easiest way to deny access to group is to never add them to the forum. For example if you had private forum for admins and moderators you would only add those groups. All other groups would be denied access by default.

Once you have added or removed groups from the selected forums you can also highlight all the groups and select edit permissions. On this page you can set permissions, *using the role as one example.

Washerman · Post by **Washerman** » Tue Feb 20, 2018 5:24 pm

thecoalman... I knew that I must be overlooking something. Sincere thanks for your help and assistance - problem solved !

advertisements