Bot Or Site Scraper?

[CGI1984]

I just noticed a new site guest, which looks like it could be a Facebook bot, but not 100% sure. This is what I saw for user info: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) - IP 173.252.87.42

If it is a Facebook bot then should I add it to my Spiders/Robots and would the User Agent be "facebot"?

Thanks.

[KYPREO]

I just noticed a new site guest, which looks like it could be a Facebook bot, but not 100% sure. This is what I saw for user info: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) - IP 173.252.87.42

If it is a Facebook bot then should I add it to my Spiders/Robots and would the User Agent be "facebot"?

Thanks.

Facebook doesn't crawl sites but this user agent is sometimes classified as a bot. It is actually the user agent when a user accesses your site via the inbuilt browser in the Facebook app. I assume there was a link to your board somewhere on Facebook.

I wouldn't add it as a bot as it is essentially a real life guest visiting via Facebook.

[CGI1984]

Thanks. I also had thought that it might have been someone following a link to our site from Facebook, but when I did a search for Facebook user agent then facebookexternalhit/* popped up on Stack Overflow on the link below and said that the Facebook User Agent is facebot:

https://stackoverflow.com/questions/862 ... user-agent

[KYPREO]

Yes after reading that and doing some of my own reading, my previous post was wrong. It looks like Facebook "crawls" the site to gather the metadata necessary to display OpenGraph properties (preview image, topic name etc) when a link is posted on Facebook, see: https://developers.facebook.com/docs/sh ... rs/crawler

This is probably I have seen these hits coincide with my board being linked on Facebook and came to the conclusion I did.

I still think it's not in the same category as other bots. The only reason you want to add specific bots like this is control permissions and to manage sessions where the bot is opening lots of concurrent connections, like a search engine crawler. This Facebook user-agent doesn't do that. It just hits pages sporadically when a page is linked on Facebook. If you don't define it as a bot, it will be treated as a guest, and that is probably appropriate in the circumstances.

[CGI1984]

Thanks again. The only other reason I see for adding it as a bot is it would make it faster for you to recognize who is online when it visits the site. I realize it isn’t a bot in the conventional sense, and only visits to checkup on a link someone added for your board to FaceBook, but when it’s on the site it is acting as a bot and collecting meta data. Since it is going to be doing the same task every time it’s on the site it will help give you an idea (at quick glance of who is online) how often FaceBook is pulling data from your site. So not a bad idea to add it to the bots I think and it’s not there acting like a normal human guest who is normally there reading posts out of personal interest.

[CGI1984]

Just saw this DuckDuckGo Favicons bot on my site. Add?

https://developers.whatismybrowser.com/ ... icons-bot/

[Lumpy Burgertushie]

as was said above, the only reason for the bot group is so you can control permissions for any and all bots in that group.

you will get hundreds of bots visiting your site, there is no reason to put them all in the bot group.

most people never touch that group and it works just fine for them.

luck,
robert

[3Di]

...

Facebook doesn't crawl sites but this user agent is sometimes classified as a bot. It is actually the user agent when a user accesses your site via the inbuilt browser in the Facebook app. I assume there was a link to your board somewhere on Facebook.

I wouldn't add it as a bot as it is essentially a real life guest visiting via Facebook.

I added it because having 12 guests at the same time with the same UA/IP didn't seem very common, and I personally have no reference to facecrook on my site, so it's not explained otherwise.

[johns1124]

For some reason over the last two weeks I have been hit by this:
phpbb facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

at the tune of over 150+ connections at one time. I'm starting to think its time to block it. via .htaccess since it used
multi-IP's

Anyone have more experience with this bot?

[bonelifer]

You could try contacting the email address on that page. But from a Google search, it seems like Facebooks bots don't respect crawl delay directive in robots.txt, so if they are hitting you that hard at times, and they don't reply, probably best to block them outright via .htaccess file.

Only thing it'll hurt is when people share your board on Facebook, it wont get the page title and images.