Indexing problem

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Indexing problem

Post by MortallCsAl3x »

hello community, I also have a phpbb3 forum and I want you to help me please with some problems.

1 It tells me something about the robots.txt file that prevents indexing, where can I find the file because I don't even have such a thing.

screenshot: https://i.postimg.cc/QdM2ZxRV/Screensho ... oid-gm.jpg

2 over 2 and a half months since the forum was online, I don't see the posts indexed on Google, like the resources that members posted, topics and so on, what should I do?
Last edited by Mick on Thu May 16, 2024 5:26 pm, edited 1 time in total.
Reason: Solved.
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Brf
Support Team Member
Support Team Member
Posts: 53538
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}

Re: Indexing problem

Post by Brf »

You need to give the Bots group permissions to see your forums
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

Last edited by MortallCsAl3x on Wed May 15, 2024 1:24 pm, edited 1 time in total.
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Brf
Support Team Member
Support Team Member
Posts: 53538
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}

Re: Indexing problem

Post by Brf »

Permissions / Group Forum permissions
Bots Group
Select all forums
Set to "Bots permissions"
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

Brf wrote: Wed May 15, 2024 1:22 pm Permissions / Group Forum permissions
Bots Group
Select all forums
Set to "Bots permissions"


https://i.postimg.cc/Gpgvtr3M/Screensho ... chrome.jpg Like here?
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Brf
Support Team Member
Support Team Member
Posts: 53538
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}

Re: Indexing problem

Post by Brf »

Yes
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

And with point 1, can someone help me, that phase with robots.txt doesn't even exist on my webhost, what should I do?
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
warmweer
Jr. Extension Validator
Posts: 11723
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Indexing problem

Post by warmweer »

MortallCsAl3x wrote: Wed May 15, 2024 6:56 pm And with point 1, can someone help me, that phase with robots.txt doesn't even exist on my webhost, what should I do?
https://developers.google.com/search/do ... bots/intro
Spelling is freeware, which means you can use it for free.
On the other hand, it is not open source, which means you cannot change it or publish it in a modified form.


Time flies like an arrow, but fruit flies like a banana.
User avatar
Forex Station
Registered User
Posts: 185
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Indexing problem

Post by Forex Station »

MortallCsAl3x wrote: Wed May 15, 2024 6:56 pm And with point 1, can someone help me, that phase with robots.txt doesn't even exist on my webhost, what should I do?
You don't actually need one to have anything in it, but just make sure you have a text file with the name robots.txt in your public folder eg. robots.txt

We haven't had a proper robots.txt for years and always rank number on the first page of search engines for most of our category's searches.
Highly-customized PhpBB board, voted as one of the most influential trading sites in the world: forex-station.com šŸ’¬
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

I make one file in public_html with robots.txt empty. So i need to add Something in to it?
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26846
Joined: Fri Aug 29, 2008 9:49 am

Re: Indexing problem

Post by Mick »

Yes, info about any bad bots that are visiting your site for example:

Code: Select all

User-Agent: PetalBot
Disallow: /
the above will stop the Petalbot bot from indexing your board but, from what I can gather, having a robots.txt file or not is not your issue, you want good bots like Google to index your board?

FWIW - Two months isnā€™t a long time to get properly indexed, sometimes it can take six months or longer, just make sure you have the settings for the bots group as previously mentioned and you should be fine, sometimes itā€™s just a matter of patience. Also, do you have lots of unique information and users regularly posting fresh content? Content is usually the key to getting indexed properly.
  • "The more connected we get the more alone we becomeā€ - Kyle BroflovskiĀ© šŸ‡¬šŸ‡§
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

Mick wrote: Thu May 16, 2024 7:13 am
yes I created that robots.txt. can you tell me exactly if I need it and what I need to add in it please?

and with the permissions part, yes, I added the access robot group permission to all the forums. See above in the picture, I think it is correct.
:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26846
Joined: Fri Aug 29, 2008 9:49 am

Re: Indexing problem

Post by Mick »

But robots.txt is nothing to do with the subject and, in many cases, is unnecessary. In fact, I never used one in 14 or so years until lately with the emergence of Claudebot and AI bots in general. But, to answer the question about robots.txt, keep an eye on ā€˜Who is onlineā€™ with ā€˜Display guestsā€™ clicked. If youā€™ve got tons of stuff in there you donā€™t understand post a screencap so we can see whatā€™s going on but I canā€™t see anybody online at all bots or humans atm. You can also control the bots in the .htaccess file, see Claudebot attack for more information on that.

As this is a separate subject you should start a new topic for that.

Back to the topic:
Mick wrote: Thu May 16, 2024 7:13 amFWIW - Two months isnā€™t a long time to get properly indexed, sometimes it can take six months or longer, just make sure you have the settings for the bots group as previously mentioned and you should be fine, sometimes itā€™s just a matter of patience. Also, do you have lots of unique information and users regularly posting fresh content? Content is usually the key to getting indexed properly.
Presumably weā€™re talking about https://www.worldcs.ro/forum?
  • "The more connected we get the more alone we becomeā€ - Kyle BroflovskiĀ© šŸ‡¬šŸ‡§
MortallCsAl3x
Registered User
Posts: 80
Joined: Fri Apr 14, 2023 5:01 pm
Location: Romania
Name: Alex Cornea

Re: Indexing problem

Post by MortallCsAl3x »

:roll: Forum - https://www.worldcs.ro - Gaming Community PHPBB3
:arrow: Counter-Strike 1.6 download - https://www.cstrike16.eu
User avatar
Brf
Support Team Member
Support Team Member
Posts: 53538
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}

Re: Indexing problem

Post by Brf »

I am seeing some viewforums on Google now since you set the permissions.

Return to ā€œ[3.3.x] Support Forumā€