Google indexing posts

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
Lumpy Burgertushie
Registered User
Posts: 66858
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Google indexing posts

Post by Lumpy Burgertushie » Wed Nov 06, 2019 12:26 am

not really. that is why sitemaps are not much good for things like a bulletin board or even a blog.

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

If a tree falls in the forest and nobody is there, does it make a sound?

KYPREO
Registered User
Posts: 123
Joined: Fri Feb 02, 2018 9:56 am
Contact:

Re: Google indexing posts

Post by KYPREO » Wed Nov 06, 2019 10:41 pm

I agree with the basic proposition that you don't need a sitemap for Google to find and index your content.

However, it is wrong to say you can't have a sitemap for a dynamic website. That is precisely what the sitemap extension does - it creates a sitemap, caches it for a user configurable period of time then rebuilds based on changes. Once you upload the sitemap location to Google Search Console, it will know where to look for the most up-to-date version of the sitemap.

In any event, Google will only crawl the same pages every few weeks, so a sitemap only needs to be updated every month or so.

Furthermore, a sitemap has other benefits besides indexing. It can be used to tell Google the canonical URL for website pages. Google will crawl the forum organically and guess the URL for each page based on how it came to that page. For example, Google will often index a page as forum.com/viewtopic.php?t=### or viewtopic.php?p=####, whereas you may wish for the link to be recognised as forum.com/viewtopic.php?f=#&t=###.

A sitemap might also help Google (and other search engines eg Bing) more efficiently. For example, I notice Google bots trying to hit user profile and search pages where I have forbidden access. These come up as crawl errors in Google Search Console. The bots are simply following URLs in the buttons for user profile etc but are repeatedly trying to hit the same pages to get a 403 error every time. If you use a sitemap, it acts as an instruction to the bots that they are not meant to access those pages. Although I have not tested it, I suspect this might reduce resources consumed by search engine bots who have access to the sitemap.

For the OP, I suggest you follow my initial advice and register for Google Search Console. This has a tool which tells you (among other things) whether Google's crawlers are having any difficulties accessing your site. There might be permission issues or temporary 500, 404 or 403 errors you're not aware of that is causing Google to slow down its crawling or refusing to crawl certain parts of your site. If there are no issues identified in Search Console, then there is nothing more you can do - it is just that Google has decided not to index certain pages for whatever reason. Keep in mind also that Google now uses its smartphone bots as its primary crawlers, so if your board is not smartphone friendly (eg non-responsive theme), then this may impact on what Google is choosing to crawl.

You've said that only 6 months of posts are indexed, but based on your responses to date I can only assume you haven't registered for Google Search Console and you have no real way of knowing to what extent your site is indexed. Google search results performance =/= Google Index. Google might index 10,000 pages on your board, but only 10% come up when you search certain keywords.

I have around 50,000 pages in Google's index but this is only a fraction of the total number of pages which are accessible to Google's bots (my board has over 160,000 topics, many of which extend to thousands of pages). From that perspective if only the last 6 months of your board is indexed, then this would not be entirely unusual. However, I can tell you that posts going back to 2002 are indexed and come up high in search results.
phpBB user since 2002
www.AusRotary.com

Post Reply

Return to “phpBB Discussion”