Are these good or bad bots?

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Post Reply
User avatar
Traction
Registered User
Posts: 345
Joined: Fri Jun 06, 2003 8:42 pm
Location: Staten Island, New York
Contact:

Are these good or bad bots?

Post by Traction »

A few weeks ago on December 21, four bots joined my forums. They are as follows:

Code: Select all

Ahrefs [Bot]
Amazon [Bot]
Semrush [Bot]
DuckDuckGo [Bot]
Are they all necessary, or should I delete one (or some/all) of them?
Failure is not an option.
Zap! Classic Video Game Forums
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Are these good or bad bots?

Post by 3Di »

Is all right there.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Traction
Registered User
Posts: 345
Joined: Fri Jun 06, 2003 8:42 pm
Location: Staten Island, New York
Contact:

Re: Are these good or bad bots?

Post by Traction »

3Di wrote: Mon Jan 10, 2022 10:29 pm Is all right there.
So keep them? Thanks!
Failure is not an option.
Zap! Classic Video Game Forums
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Are these good or bad bots?

Post by thecoalman »

The bot group in phpBB helps you identify known bots based on their user agent string, you can give them different permissions and phpBB also makes minor changes to the pages. For example usernames are no longer linked to the profile. Removing them doesn't accomplish anything because then they are anonymous users. This would be counter productive because pages are going to have links to resources you do not want any bot accessing. For example if bot/guest tries to access the profile page they will get a "You do not have permission..." error page, it's just a waste of server resources and removing those links prevents bots from trying to follow them in the first place.

If you want better control over legitimate bots look into using robots.txt, typically if they are identifying themselves with user agent string they will adhere to robots.txt directives. They can be blocked entirely with robots.txt... keep in mind what I said was "legitimate bots". The ahrefs and semrush bots are marketing bots and can be denied access through robots.txt, for most sites they serve no useful purpose. Not sure about Amazon but DuckDuckGo is legitimate search bot, their search focuses on user privacy. You can of course research each bot and see what it's for.

Last but not least be aware the user agent can be spoofed. For example there is browser extensions to switch to user agent to Google bot and others. You can then brows your forum to see what the pages are like as Google sees them.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Are these good or bad bots?

Post by 3Di »

Traction wrote: Mon Jan 10, 2022 11:49 pm
3Di wrote: Mon Jan 10, 2022 10:29 pm Is all right there.
So keep them? Thanks!
Yeah, no probs.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
Post Reply

Return to “phpBB Discussion”