Google is not crawling to individual topic or post level immediately

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
4mpb5
Registered User
Posts: 151
Joined: Wed Sep 08, 2021 9:34 pm

Google is not crawling to individual topic or post level immediately

Post by 4mpb5 »

Google sees my forum site at the top level with description.
But it is not crawling down to individual post or topic level immediately

I see individuall posts made to other phpBB forums show up within minutes on Google.

I do see Google listed as an authorized crawler in my forum.

So what I am doing wrong ?
It seems to take weeks or months before the posts on my forum come up on Google.

BTW are there any other (major) search engines other than Google on internet ?
All other search engines like evn Yahoo seem to crawl off of Google ?

There may be some in China etc but not in English language ?

What free sites do you recommend to list my forum, for maximum visibility on the internet globally ?
User avatar
AbuHossam
Registered User
Posts: 361
Joined: Sun Sep 02, 2018 7:24 am

Re: Google is not crawling to individual topic or post level immediately

Post by AbuHossam »

Indeed, it may require some patience and valuable content that Google can recognize.

However, let me assure you that Google now considers forums, especially phpBB, to be of lower value.

As you can see from my dashboard, this is the case right now.
image f.jpg
Therefore, don't lose hope and keep working hard.
You do not have the required permissions to view the files attached to this post.
Recent Topics in DB-UK Community – Where Every Passion Finds Its Place.
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26829
Joined: Fri Aug 29, 2008 9:49 am

Re: Google is not crawling to individual topic or post level immediately

Post by Mick »

AbuHossam wrote: Fri Mar 29, 2024 8:54 amespecially phpBB
Who says so?
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧
User avatar
AbuHossam
Registered User
Posts: 361
Joined: Sun Sep 02, 2018 7:24 am

Re: Google is not crawling to individual topic or post level immediately

Post by AbuHossam »

Mick wrote: Fri Mar 29, 2024 9:12 am Who says so?
Based on my personal experience and what others have said, I initially found it hard to believe that all websites are treated equally by Google.

However, after conducting my own experiments, I realized that phpBB forums indeed require significant effort and work to gain the trust of search engines.
Recent Topics in DB-UK Community – Where Every Passion Finds Its Place.
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26829
Joined: Fri Aug 29, 2008 9:49 am

Re: Google is not crawling to individual topic or post level immediately

Post by Mick »

So this is your opinion?
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧
User avatar
AbuHossam
Registered User
Posts: 361
Joined: Sun Sep 02, 2018 7:24 am

Re: Google is not crawling to individual topic or post level immediately

Post by AbuHossam »

Mick wrote: Fri Mar 29, 2024 9:23 am So this is your opinion?
If you see it that way/want to see it that way, then yes, it is.
Recent Topics in DB-UK Community – Where Every Passion Finds Its Place.
dextrous
Registered User
Posts: 23
Joined: Fri Mar 01, 2024 2:52 pm
Name: Dextrous

Re: Google is not crawling to individual topic or post level immediately

Post by dextrous »

Our understanding is that there are lot of factors google considers when it ranks Domain & Pages, bottom line is content has to be valuable & there should not be lot of clutters. Manipulating/Spamming impacts the ranking negatively, it takes lot of time & patience to push the rank. I am not sure about the PHPBB or any other forum not being scanned by google, don't think that is true. Because google has a separate section "Discussions and forums" on the top 2nd or 3rd search results based on the user search words, so if your forum has all the words & intent then google brings up the page on the top.
Dextrous
Deals TeamUpTalk Website: https://deals.teamuptalk.com
User avatar
Forex Station
Registered User
Posts: 185
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Google is not crawling to individual topic or post level immediately

Post by Forex Station »

4mpb5 wrote: Thu Mar 28, 2024 5:38 pm Google sees my forum site at the top level with description.
But it is not crawling down to individual post or topic level immediately...
Our forum's latest posts and files get indexed every hour or two.

How old is your forum? Reason I ask is when my forum was under a year old I had it on the default BOTS permissions and our site barely got shown in search results and I found it a huge struggle to get topics indexed. After adjusting the Google [BOT]'s permissions in the ACP to allow it to ignore flood limit and give it access to search the forum, see profiles etc. that's when I started seeing lots of our pages indexed.

By leaving those settings over the past 7 years we have almost 200,000 pages of our site indexed.

(The recent drop was due to me experimenting with blocking certain pages (like "Thanks For Posts" list) using robots.txt)
GSC.png
You do not have the required permissions to view the files attached to this post.
Highly-customized PhpBB board, voted as one of the most influential trading sites in the world: forex-station.com 💬
dextrous
Registered User
Posts: 23
Joined: Fri Mar 01, 2024 2:52 pm
Name: Dextrous

Re: Google is not crawling to individual topic or post level immediately

Post by dextrous »

Forex Station wrote: Sat Mar 30, 2024 2:15 pm
Do you just leave all 200,000 pages in the same forum & sitemap.xml?. Or do you move these into archive?. Thanks.
Last edited by Mick on Mon Apr 15, 2024 7:09 am, edited 1 time in total.
Reason: Removed unnecessary full quotes.
Dextrous
Deals TeamUpTalk Website: https://deals.teamuptalk.com
User avatar
Forex Station
Registered User
Posts: 185
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Google is not crawling to individual topic or post level immediately

Post by Forex Station »

dextrous wrote: Tue Apr 02, 2024 8:13 pm
Sorry I'm not sure what you mean by "archive".

I don't use sitemaps anymore. I use to generate sitemaps using Screaming Frog for creating sitemaps but only saw a small boost in SEO and impressions so I stopped using sitemaps altogether.
Last edited by Mick on Mon Apr 15, 2024 7:10 am, edited 1 time in total.
Reason: Removed unnecessary full quotes.
Highly-customized PhpBB board, voted as one of the most influential trading sites in the world: forex-station.com 💬
dextrous
Registered User
Posts: 23
Joined: Fri Mar 01, 2024 2:52 pm
Name: Dextrous

Re: Google is not crawling to individual topic or post level immediately

Post by dextrous »

Ok, so you still have all these 200K topics sitting in the same forum?.
Dextrous
Deals TeamUpTalk Website: https://deals.teamuptalk.com
User avatar
JLA
Registered User
Posts: 618
Joined: Tue Nov 16, 2004 5:23 pm
Location: USA
Name: JLA FORUMS

Re: Google is not crawling to individual topic or post level immediately

Post by JLA »

Be sure that your link rel= tags are correct on your pages (you can see example on our site pages) as well as your structured data markup.

As others mentioned - general site maps too.

Return to “phpBB Discussion”