Page 1 of 1

Disable the ?sid= from url for google seo spider

Posted: Sat Jun 01, 2019 2:56 pm
by eselva
Hello, im trying to optimize my forum for seo purposes but I've noticed that the bots take the same url repeated hundreds of times because of the variable sid, how can i disable it? I tried to disallow the sid on robots.txt, but then it disallows the whole /viewtopic.php urls . I think my cookies config on the adm are correct.

Re: Disable the ?sid= from url for google seo spider

Posted: Sat Jun 01, 2019 11:30 pm
by stevemaury
Post a link to your board and post your cookie settings. Also, is GoogleBOT in the Bots group?

Re: Disable the ?sid= from url for google seo spider

Posted: Sun Jun 02, 2019 11:40 pm
by eselva
stevemaury wrote:
Sat Jun 01, 2019 11:30 pm
Post a link to your board and post your cookie settings. Also, is GoogleBOT in the Bots group?
Thanks for the reply!

Yes, it is. But im using semrush to emulate google crawler and i just noticed that the semrush bot is not there. I will add it and then check if that was the problem...


https://blaugranas.es/

Image

Re: Disable the ?sid= from url for google seo spider

Posted: Sun Jun 02, 2019 11:54 pm
by Brf
When I look up your site on Google, I am not seeing any sids in any of the URLs.

Re: Disable the ?sid= from url for google seo spider

Posted: Mon Jun 03, 2019 1:57 am
by thecoalman
Brf wrote:
Sun Jun 02, 2019 11:54 pm
When I look up your site on Google, I am not seeing any sids in any of the URLs.
They would be tagged as duplicate content and not appear in the search results if they were already indexed.

Re: Disable the ?sid= from url for google seo spider

Posted: Mon Jun 03, 2019 9:01 am
by eselva
thecoalman wrote:
Mon Jun 03, 2019 1:57 am
Brf wrote:
Sun Jun 02, 2019 11:54 pm
When I look up your site on Google, I am not seeing any sids in any of the URLs.
They would be tagged as duplicate content and not appear in the search results if they were already indexed.
stevemaury wrote:
Sat Jun 01, 2019 11:30 pm
Post a link to your board and post your cookie settings. Also, is GoogleBOT in the Bots group?
Brf wrote:
Sun Jun 02, 2019 11:54 pm
When I look up your site on Google, I am not seeing any sids in any of the URLs.
Thanks for replies, I solved this issue now. From the beginning wasn't a problem with google, but with the bot I used to simulate the google bot( semrush). If you don't add it to the bot list, logically it takes sid in the url, in this case semrush uses 2 bots and multiplied the whole site x2 . Sorry, that was easy to solve :roll: .

There is the actual configuration if you want to use semrush to optimize your phpbb seo (im not sure if im able to post here the ip credentials, remove it if not):

SEMrush 66
SemrushBot
46.229.173.66

-
SEMrush 68
SemrushBot
46.229.173.68