AhrefsBot

Deactivated 1950056 · Post by **Deactivated 1950056** » Fri Sep 18, 2020 5:03 pm

On a board I'm an Admin on, we keep on getting visited by something called the AhrefsBot.
It has the following agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Have you guys had any visits from this bot, and would you say its a good bot or a bad one? I find it interesting that it shows up as a Guest and it is active pretty much constantly. It now shows up in the online list as more than one user, so I worry if it's going to cause problems.

Edit: My apologies if this is in the wrong forum.

Post by **P_I** » Sat Sep 19, 2020 1:51 pm

I consider it a bad, resourcing hogging bot so long ago blocked it in robots.txt.

Post by **thecoalman** » Sun Sep 20, 2020 5:01 am

It's a bot used by marketing industry. Worthless for most sites. It may listen to robots.txt

User-agent: AhrefsBot
Disallow: /

Quick tip, you can add bots to other groups. Create a group called bad bot and ad the bot, set permissions to never for everything.

As another example if you had forum you were displaying Google ads but denying bots because you didn't want it indexed. Create special group for Googles ad bot, give it permission to access. One caveat here, if the forum was behind login someone could access it by switching user agent. You wouldn't use this method if you needed to keep guests out.

Deactivated 1950056 · Post by **Deactivated 1950056** » Sun Sep 20, 2020 5:13 am

thecoalman wrote: ↑Sun Sep 20, 2020 5:01 amQuick tip, you can add bots to other groups. Create a group called bad bot and ad the bot, set permissions to never for everything.

Thanks for the robots.text line.

BTW, I have tried to add it as a bot in phpBB (it always shows up as a Guest), but that did not work for some reason, and I have a feeling that was intentional on the part of the bot's creator (though I may be wrong.)

Post by **thecoalman** » Sun Sep 20, 2020 11:16 am

Try clearing the cache after adding it, they may be cached.

JLA · Post by **JLA** » Sun Sep 20, 2020 7:55 pm

Random American wrote: ↑Fri Sep 18, 2020 5:03 pm On a board I'm an Admin on, we keep on getting visited by something called the AhrefsBot.
It has the following agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Have you guys had any visits from this bot, and would you say its a good bot or a bad one? I find it interesting that it shows up as a Guest and it is active pretty much constantly. It now shows up in the online list as more than one user, so I worry if it's going to cause problems.

Edit: My apologies if this is in the wrong forum.

Block this bot. if you take a deep look at all of your site's traffic you will find many many more to block too. From one of our other posts the other day

Having been at PHPBB for over 15 years now - best thing we recommend to remember is there is not a single magic bullet to resolve malicious activity against your board. Most board owners are not fully aware of the tremendous amount of activity on/against their sites (even smaller boards) that offers absolutely no benefit to the board owner. This activity ranges from script kiddies to well funded entities who's sole purpose is to enrich themselves off of your efforts.

Protecting your board is like a really big onion. You have to have many many layers of protection and you need to make yourself aware of all of the activity on your site and it's purpose. Starting down this path is a real eye opener for most board owners.

Deactivated 1950056 · Post by **Deactivated 1950056** » Mon Sep 21, 2020 7:01 pm

I don't own the site, just an Admin, but the site owner has just blocked it with robots.txt. I still see it online on the who is online section, and it still shows up as more than one guest, but they only look at the index page now. Is that better?

Thanks.

Post by **thecoalman** » Mon Sep 21, 2020 10:05 pm

robots.txt is only occasionally downloaded, you would need to check server access logs to see if it downloaded updated version or just wait. It should do it in a few days.

As far as the online listing it shouldn't be listed as guest if you added to bots group. Make sure you didn't copy and paste a space. Once again try clearing the cache. I don't know if clearing cache has any affect but that is something that would be cacheable.

L&R · Post by **L&R** » Tue Sep 22, 2020 7:45 am

These "marketing" bots are resource wasters for us, I could add Awario to the same list. Add them to block list in robots.txt along with rules for no access in .htaccess.

Forex Station · Post by **Forex Station** » Tue Sep 22, 2020 12:38 pm

Unless you subscribe to Ahrefs, you should block it. This thing will hog resources with absolutely no benefit to you. For us, Ahrefs would take 2GB bandwidth per day on our site. We stopped that right at the .htaccess level and our Pagespeed Insights improved by 11% alone.

Deactivated 1950056 · Post by **Deactivated 1950056** » Tue Sep 22, 2020 4:23 pm

thecoalman wrote: ↑Mon Sep 21, 2020 10:05 pm robots.txt is only occasionally downloaded, you would need to check server access logs to see if it downloaded updated version or just wait. It should do it in a few days.

As far as the online listing it shouldn't be listed as guest if you added to bots group. Make sure you didn't copy and paste a space. Once again try clearing the cache. I don't know if clearing cache has any affect but that is something that would be cacheable.

I was going to make another attempt at adding them as a bot, but I no longer see them in the online list at all now. That robots.txt line seems to have worked. Thanks again for the line, I think this problem is solved.