Google Issues

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
Post Reply
User avatar
CGI1984
Registered User
Posts: 157
Joined: Thu Feb 20, 2020 8:27 am

Google Issues

Post by CGI1984 »

We are having some issues with Google.

1 - We don't want them indexing images from our site so we put something in our robots.txt file to stop them (see code below), but all images posted on our board seem to be showing up in Google Images anyway. Any ideas why this code doesn’t seem to be working?

Code: Select all

User-agent: Googlebot-Image
Disallow: / 
2 - We also added a new forum recently which we don't want Google to crawl and index and we set Bot permissions to No Access on this new forum, but we just got an email from Google Search Console saying the following in regards to the new forum we added and the posts in that forum. But they shouldn't be crawling this forum anyway so I don't understand why it’s triggered a Google crawl error. I though having no Bot access on this forum would prevent Google from even trying to crawl this forum?
Coverage issues detected.

Search Console has identified that your site is affected by 1 Coverage issues:

Top Errors

Errors can prevent your page or feature from appearing in Search results. The following errors were found on your site:

Submitted URL has crawl issue

We recommend that you fix these issues when possible to enable the best experience and coverage in Google Search.

TechieGeek
Registered User
Posts: 14
Joined: Mon Jan 11, 2010 10:33 pm
Contact:

Re: Google Issues

Post by TechieGeek »

1) This is correct. How long have you had the robots.txt live? Try this tester to see if your robots.txt is getting read correctly: https://support.google.com/webmasters/a ... 2598?hl=en

2) Perhaps Googlebot can see the link to the forum on the homepage, but upon accessing the link, it hits an error due to the lack of permission? (This is how a guest would encounter a site with a private forum as well.) You can treat the warning in the Google Search Console as an FYI - it's telling you that this page can't be crawled, but you don't want the page crawled anyhow, so no biggie. I don't believe there's a way to acknowledge the notification in the search console.
I manage a home improvement forum on phpBB.

User avatar
CGI1984
Registered User
Posts: 157
Joined: Thu Feb 20, 2020 8:27 am

Re: Google Issues

Post by CGI1984 »

Thanks for your feedback.

1 - I have had the robots.txt file live for a couple of months. I tried the tester and it says it is working fine. But Google still seems to be ignoring the Googlebot-Image - Disallow: / command and it is still continuing to index more and more images from our site on an ongoing basis. That is why I posted the issue.

2 - The Bot No Access permission setting is a standard setting under forum permissions. Normally it is used if you have private forums that you don't want indexed by search engines. We have about 5 of these No Access forums already in place that we had set up on the site since the very beginning. And we never got any warnings from Google that the forums can't be crawled. So it was never a problem before and they just ignored them. Then we recently added a new private forum which we also don't want crawled and suddenly Google is flagging that as an error. Normally Google understands the No Access setting and never raised an issue with it. But as you said, I can just ignore the warning in the Google Search Console about it, but it is confusing why this would be triggering a warning in the first place when it is something Google seems to understand and it has never caused an issue in the past.

The only thing I can think of is we are running an extension called "phpBB3 SEO Sitemap" (https://www.phpbb.com/customise/db/exte ... o_sitemap/) which creates dynamic sitemaps. Perhaps this extension has included this new "don't crawl" forum in the sitemap when it shouldn't be and this is causing confusion with the Google bot and triggering the error? I don't know, its just a guess.

User avatar
Lumpy Burgertushie
Registered User
Posts: 67740
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Google Issues

Post by Lumpy Burgertushie »

sounds logical. easy way to test is to disable the SEO sitemap extension for a while and see what happens.
also, usually the best way to find out why google does something is to ask google. nobody but google really knows what they are doing and why.

luck,
robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?

User avatar
CGI1984
Registered User
Posts: 157
Joined: Thu Feb 20, 2020 8:27 am

Re: Google Issues

Post by CGI1984 »

Thanks. I would love to be able to ask Google some things directly (not via a forum). Is there a way to do that AFAYK?

User avatar
Lumpy Burgertushie
Registered User
Posts: 67740
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Google Issues

Post by Lumpy Burgertushie »

I have no idea how google works for asking questions. I have never had a need to ask them anything.
I don't have any boards that I worry about search engine stuff. however, I have studied SEO for several years.
what I have found is that phpBB needs very little help with SEO out of the box and that google knows more about google than we do. ;)

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?

Post Reply

Return to “phpBB Discussion”