AhrefsBot

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Post Reply
User avatar
Random American
Registered User
Posts: 176
Joined: Sat Aug 10, 2019 4:45 am
Location: Somewhere in the Southern USA.

AhrefsBot

Post by Random American »

On a board I'm an Admin on, we keep on getting visited by something called the AhrefsBot.
It has the following agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Have you guys had any visits from this bot, and would you say its a good bot or a bad one? I find it interesting that it shows up as a Guest and it is active pretty much constantly. It now shows up in the online list as more than one user, so I worry if it's going to cause problems.

Edit: My apologies if this is in the wrong forum.
I'm just a regular member of the phpBB Community. I do NOT represent phpBB.com in any capacity and my opinions are solely my own.
User avatar
P_I
Registered User
Posts: 1323
Joined: Tue Mar 01, 2011 8:35 pm
Location: Staying home - Western Canada
Contact:

Re: AhrefsBot

Post by P_I »

I consider it a bad, resourcing hogging bot so long ago blocked it in robots.txt.
Last edited by Mick on Sun Sep 20, 2020 7:48 am, edited 1 time in total.
Reason: Removed unnecessary full quoting.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4253
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: AhrefsBot

Post by thecoalman »

It's a bot used by marketing industry. Worthless for most sites. It may listen to robots.txt
User-agent: AhrefsBot
Disallow: /
Quick tip, you can add bots to other groups. Create a group called bad bot and ad the bot, set permissions to never for everything.

As another example if you had forum you were displaying Google ads but denying bots because you didn't want it indexed. Create special group for Googles ad bot, give it permission to access. One caveat here, if the forum was behind login someone could access it by switching user agent. You wouldn't use this method if you needed to keep guests out.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
Random American
Registered User
Posts: 176
Joined: Sat Aug 10, 2019 4:45 am
Location: Somewhere in the Southern USA.

Re: AhrefsBot

Post by Random American »

thecoalman wrote:
Sun Sep 20, 2020 5:01 am
Quick tip, you can add bots to other groups. Create a group called bad bot and ad the bot, set permissions to never for everything.
Thanks for the robots.text line.

BTW, I have tried to add it as a bot in phpBB (it always shows up as a Guest), but that did not work for some reason, and I have a feeling that was intentional on the part of the bot's creator (though I may be wrong.)
I'm just a regular member of the phpBB Community. I do NOT represent phpBB.com in any capacity and my opinions are solely my own.
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4253
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: AhrefsBot

Post by thecoalman »

Try clearing the cache after adding it, they may be cached.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
JLA
Registered User
Posts: 589
Joined: Tue Nov 16, 2004 5:23 pm
Location: USA
Name: JLA FORUMS
Contact:

Re: AhrefsBot

Post by JLA »

Random American wrote:
Fri Sep 18, 2020 5:03 pm
On a board I'm an Admin on, we keep on getting visited by something called the AhrefsBot.
It has the following agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Have you guys had any visits from this bot, and would you say its a good bot or a bad one? I find it interesting that it shows up as a Guest and it is active pretty much constantly. It now shows up in the online list as more than one user, so I worry if it's going to cause problems.

Edit: My apologies if this is in the wrong forum.
Block this bot. if you take a deep look at all of your site's traffic you will find many many more to block too. From one of our other posts the other day
Having been at PHPBB for over 15 years now - best thing we recommend to remember is there is not a single magic bullet to resolve malicious activity against your board. Most board owners are not fully aware of the tremendous amount of activity on/against their sites (even smaller boards) that offers absolutely no benefit to the board owner. This activity ranges from script kiddies to well funded entities who's sole purpose is to enrich themselves off of your efforts.

Protecting your board is like a really big onion. You have to have many many layers of protection and you need to make yourself aware of all of the activity on your site and it's purpose. Starting down this path is a real eye opener for most board owners.
User avatar
Random American
Registered User
Posts: 176
Joined: Sat Aug 10, 2019 4:45 am
Location: Somewhere in the Southern USA.

Re: AhrefsBot

Post by Random American »

I don't own the site, just an Admin, but the site owner has just blocked it with robots.txt. I still see it online on the who is online section, and it still shows up as more than one guest, but they only look at the index page now. Is that better?

Thanks.
I'm just a regular member of the phpBB Community. I do NOT represent phpBB.com in any capacity and my opinions are solely my own.
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 4253
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: AhrefsBot

Post by thecoalman »

robots.txt is only occasionally downloaded, you would need to check server access logs to see if it downloaded updated version or just wait. It should do it in a few days.

As far as the online listing it shouldn't be listed as guest if you added to bots group. Make sure you didn't copy and paste a space. Once again try clearing the cache. I don't know if clearing cache has any affect but that is something that would be cacheable.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
User avatar
L&R
Registered User
Posts: 45
Joined: Mon Oct 08, 2018 5:47 am

Re: AhrefsBot

Post by L&R »

These "marketing" bots are resource wasters for us, I could add Awario to the same list. Add them to block list in robots.txt along with rules for no access in .htaccess.
User avatar
Forex Station
Registered User
Posts: 114
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: AhrefsBot

Post by Forex Station »

Unless you subscribe to Ahrefs, you should block it. This thing will hog resources with absolutely no benefit to you. For us, Ahrefs would take 2GB bandwidth per day on our site. We stopped that right at the .htaccess level and our Pagespeed Insights improved by 11% alone.
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
User avatar
Random American
Registered User
Posts: 176
Joined: Sat Aug 10, 2019 4:45 am
Location: Somewhere in the Southern USA.

Re: AhrefsBot

Post by Random American »

thecoalman wrote:
Mon Sep 21, 2020 10:05 pm
robots.txt is only occasionally downloaded, you would need to check server access logs to see if it downloaded updated version or just wait. It should do it in a few days.

As far as the online listing it shouldn't be listed as guest if you added to bots group. Make sure you didn't copy and paste a space. Once again try clearing the cache. I don't know if clearing cache has any affect but that is something that would be cacheable.
I was going to make another attempt at adding them as a bot, but I no longer see them in the online list at all now. That robots.txt line seems to have worked. Thanks again for the line, I think this problem is solved.
I'm just a regular member of the phpBB Community. I do NOT represent phpBB.com in any capacity and my opinions are solely my own.
Post Reply

Return to “phpBB Discussion”