Preventing Bots From Indexing Certain Forums

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
Post Reply
User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Preventing Bots From Indexing Certain Forums

Post by ChinaGal »

We have a site that will have a few hidden forums on it that we will only given access to certain users on a case by case basis using permissions. So we would like to block all bots from indexing the content of these particular forums. I think I have done it right, but just want to please check to make sure I have done this correctly so there are no mistakes.

First I went into ACP and then into Forum Permissions and removed bots from having permissions for all forums and sub-forums that we want to prevent from being indexed by any search engines.

Then I also created a Robots.txt file and inserted it into our server's root directory where the forum is located and in the Robots.txt file I put the following code for example:

Code: Select all

User-agent: * 
Disallow: https://www.myforum.com/viewforum.php?f=38
Disallow: https://www.myforum.com/viewforum.php?f=39
Disallow: https://www.myforum.com/viewforum.php?f=40
Disallow: https://www.myforum.com/viewforum.php?f=41
Can someone please kindly let me know if the above steps taken are correct or if anything else would need to be done to prevent search engines from indexing these hidden forums on our site?

Thank you kindly.

KYPREO
Registered User
Posts: 312
Joined: Fri Feb 02, 2018 9:56 am
Contact:

Re: Preventing Bots From Indexing Certain Forums

Post by KYPREO »

The robots.txt entries are unnecessary and if anything act as a honeypot for bad bots to gather as a shopping list for sections you don't want them to try to access ;) Another potential issue with trying to block viewtopic is that, depending on the bot, viewtopic.php?f=2 might also signal good bots not to crawl ?f=20, f=21, f=22 etc when you might otherwise want them to.

It is sufficient not to allow bots permissions to view the hidden forums.

If you want to block hits to those sections altogether, you can use page rules on Cloudflare to block access to specific pages at the domain firewall level, but I don't think it is necessary.
phpBB user since 2002
www.AusRotary.com

User avatar
david63
Registered User
Posts: 17247
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Preventing Bots From Indexing Certain Forums

Post by david63 »

ChinaGal wrote:
Tue Feb 04, 2020 5:36 am
I went into ACP and then into Forum Permissions and removed bots from having permissions for all forums and sub-forums
Not quite sure what you have dome there. You should have given the Bots group "No access" to the fora that you do not want indexing. If that is what you did then there is nothing else that you need to do.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Preventing Bots From Indexing Certain Forums

Post by ChinaGal »

KYPREO wrote:
Tue Feb 04, 2020 6:20 am
The robots.txt entries are unnecessary and if anything act as a honeypot for bad bots to gather as a shopping list for sections you don't want them to try to access ;) Another potential issue with trying to block viewtopic is that, depending on the bot, viewtopic.php?f=2 might also signal good bots not to crawl ?f=20, f=21, f=22 etc when you might otherwise want them to.

It is sufficient not to allow bots permissions to view the hidden forums.

If you want to block hits to those sections altogether, you can use page rules on Cloudflare to block access to specific pages at the domain firewall level, but I don't think it is necessary.
Thank you very much. So you are suggesting it is best to just delete the Robots.txt file at this point?

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Preventing Bots From Indexing Certain Forums

Post by ChinaGal »

david63 wrote:
Tue Feb 04, 2020 6:22 am
ChinaGal wrote:
Tue Feb 04, 2020 5:36 am
I went into ACP and then into Forum Permissions and removed bots from having permissions for all forums and sub-forums
Not quite sure what you have dome there. You should have given the Bots group "No access" to the fora that you do not want indexing. If that is what you did then there is nothing else that you need to do.
Sorry, yes, I realize now what I wrote was a bit confusing. I meant that I went into Forum Permissions and removed bots from having permissions for the particular forum that I don't want to be indexed. I say "removed" since bots are by default given access to all forums. So I think I did it correctly. Below is a screen shot of what I did:
Screen.jpg
.

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Preventing Bots From Indexing Certain Forums

Post by ChinaGal »

I just want to add that hopefully I did everything correctly now and if I need to do anything further on this then I would kindly appreciate any additional feedback.

User avatar
david63
Registered User
Posts: 17247
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Preventing Bots From Indexing Certain Forums

Post by david63 »

It is safer to give the Bots no access than to remove them as you have done
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
ChinaGal
Registered User
Posts: 90
Joined: Fri Jan 24, 2020 10:02 am

Re: Preventing Bots From Indexing Certain Forums

Post by ChinaGal »

david63 wrote:
Tue Feb 04, 2020 7:35 am
It is safer to give the Bots no access than to remove them as you have done
Well noted and thank you. Have done that now.

User avatar
enerahed
Registered User
Posts: 7
Joined: Fri Jan 31, 2020 5:04 am

Re: Preventing Bots From Indexing Certain Forums

Post by enerahed »

Thanks for the information.
Does set a password for the forum could be done?

User avatar
KevC
Support Team Member
Support Team Member
Posts: 69625
Joined: Fri Jun 04, 2004 10:44 am
Location: Oxford, UK
Contact:

Re: Preventing Bots From Indexing Certain Forums

Post by KevC »

enerahed wrote:
Tue Feb 04, 2020 6:19 pm
Thanks for the information.
Does set a password for the forum could be done?
Yes but that inconveniences your users as well. Permissions are much simpler. Set the bots to no access and that's it.
-:|:- Support Request Template -:|:-
Image
Cheap UK Hosting
"In the land of the blind the little green bloke with no pupils is king - init!"

User avatar
enerahed
Registered User
Posts: 7
Joined: Fri Jan 31, 2020 5:04 am

Re: Preventing Bots From Indexing Certain Forums

Post by enerahed »

Thanks a lot @KevC.

User avatar
Mick
Support Team Member
Support Team Member
Posts: 22083
Joined: Fri Aug 29, 2008 9:49 am
Location: Watching cricket.

Re: Preventing Bots From Indexing Certain Forums

Post by Mick »

@enerahed: In future, if you have questions please start your own topics, thanks.
"The more connected we get the more alone we become" - Kyle Broflovski

Please read: “Am I In The Right Place?” before posting.

User avatar
enerahed
Registered User
Posts: 7
Joined: Fri Jan 31, 2020 5:04 am

Re: Preventing Bots From Indexing Certain Forums

Post by enerahed »

Mick wrote:
Wed Feb 05, 2020 9:16 am
@enerahed: In future, if you have questions please start your own topics, thanks.
I am sorry for that but I can not start a new thread. Maybe because I am still new member with few posts.
Apologized.

User avatar
KevC
Support Team Member
Support Team Member
Posts: 69625
Joined: Fri Jun 04, 2004 10:44 am
Location: Oxford, UK
Contact:

Re: Preventing Bots From Indexing Certain Forums

Post by KevC »

enerahed wrote:
Wed Feb 05, 2020 11:52 am
Mick wrote:
Wed Feb 05, 2020 9:16 am
@enerahed: In future, if you have questions please start your own topics, thanks.
I am sorry for that but I can not start a new thread. Maybe because I am still new member with few posts.
Apologized.
You definitely should be able to. Brand new members can start a topic with their first post within minutes of registering.
-:|:- Support Request Template -:|:-
Image
Cheap UK Hosting
"In the land of the blind the little green bloke with no pupils is king - init!"

User avatar
enerahed
Registered User
Posts: 7
Joined: Fri Jan 31, 2020 5:04 am

Re: Preventing Bots From Indexing Certain Forums

Post by enerahed »

KevC wrote:
Wed Feb 05, 2020 11:53 am
enerahed wrote:
Wed Feb 05, 2020 11:52 am
Mick wrote:
Wed Feb 05, 2020 9:16 am
@enerahed: In future, if you have questions please start your own topics, thanks.
I am sorry for that but I can not start a new thread. Maybe because I am still new member with few posts.
Apologized.
You definitely should be able to. Brand new members can start a topic with their first post within minutes of registering.
I might be mistaken KevC.
Apologized. :oops:

Post Reply

Return to “[3.3.x] Support Forum”