Bot Or Site Scraper?

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Bot Or Site Scraper?

Post by CGI1984 »

I noticed this Guest on my site for days going through various posts. Is this some sort of bot or maybe a site scraper copying all the site content? It shows the following as the Guest details:

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])

I don't think I should post the IP because it would be unfair to the user's privacy until I am sure that it is something malicious. I guess it could be a legitimate Guest, but it seems peculiar. Usually it shows info about the browser being used. Never seen something like that for the browser details and not sure what it is. Moz.com says it is SEO software and data to help you increase traffic, rankings, and visibility in search results. But why would it be spending lots of time on our site? We didn't hire them.

UPDATE:

I have had one of our admins block the IP until we figure out what this is. If we find out later it is a legitimate user then we will remove the block.
Last edited by CGI1984 on Sat Feb 22, 2020 9:48 pm, edited 1 time in total.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Thanks. Still not sure what they are doing. Is this for their own commercial purposes or do they contribute to search engine indexing? I figure if they are a legitimate bot then they would show up in the list of bots in ACP, no?
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

Yes it is legitimate IMHO. You can add it to the list if you so wish.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Thanks. So I should remove the IP block on it I guess?

When you sat add it to the list, you mean my list of Bots in ACP? If so, how do I do that?
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

Yessure remove it.

ACP / System / Spiders and robots / Add bot (you'll find a button at the bottom of that page)
2020-02-22 22_59_51-Spiders_Robots.png

Use a Bot name of your choice and add the UA (Agent match === dotbot), that's it. IP is not relevant.
You do not have the required permissions to view the files attached to this post.
Last edited by 3Di on Sat Feb 22, 2020 10:03 pm, edited 1 time in total.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

OK, I added it to my site.

For Agent Match I put: http://www.opensiteexplorer.org/dotbot

And for IP: 216.244.66.241 - Which is the one it has been using.

Let's see if it comes back now. Maybe it won't after being blocked for a little while.
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Did I put the right agent?
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

No IP as I said.

(Agent match === dotbot) and that's it.
2020-02-22 23_04_16-Moz's Web Crawler DotBot - Help Hub - Moz.png
You do not have the required permissions to view the files attached to this post.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Ok, thanks. Fixed. And now it's back already. Showing as a bot now. Persistent little bugger! :D
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Maybe it should be added as part of the list of bots on the default install on phpBB in the future?
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

Guess why the Add bot feature has been made for?
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

For ordering a nice cold Hefeweizen? 8-)

Thanks. No worries.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco

Re: Bot Or Site Scraper?

Post by 3Di »

CGI1984 wrote: Sat Feb 22, 2020 10:12 pm For ordering a nice cold Hefeweizen? 8-)

Thanks. No worries.
You're in close danger of running out of answers. 8-) Bitte. :)
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
CGI1984
Registered User
Posts: 203
Joined: Thu Feb 20, 2020 8:27 am

Re: Bot Or Site Scraper?

Post by CGI1984 »

Noted. Danka.Tschüss. :lol:

Return to “phpBB Discussion”