Adsbot (Google) access?

Get help with installation and running phpBB 3.0.x here. Please do not post bug reports, feature requests, or MOD-related questions here.
Anti-Spam Guide
Forum rules
END OF SUPPORT: 1 January 2017 (announcement)
Locked
asinshesq
Registered User
Posts: 6266
Joined: Sun Feb 22, 2004 9:34 pm
Location: NYC
Name: Alan

Adsbot (Google) access?

Post by asinshesq »

I just noticed adsbot (Google) online on my forum and when I clicked the adsbot link in who is online it said adsbot was reading a certain forum. This surprised me because: (1) I have a robots.txt file in the root phpbb folder that reads
User-agent: *
Disallow: /

and

(2) I deny access to all forums on my site except for registered users and I have not granted access to any bots.

What does it actually mean that adsbot was reported as online and reading a restricted forum? Am I better off deactivating all bots in the manage spiders area of the acp?
User avatar
5hocK
Registered User
Posts: 3011
Joined: Wed Nov 23, 2011 7:00 pm
Location: England

Re: Adsbot (Google) access?

Post by 5hocK »

It's not what it seems, read this: docs/FAQ.html#viewonline
Same for bots and users. As long as forum permissions for bots are set to No Access for the forum/s in question then they aren't browsing them.
A thread is pretty much a topic. Stop with the pedantics will ya.
asinshesq
Registered User
Posts: 6266
Joined: Sun Feb 22, 2004 9:34 pm
Location: NYC
Name: Alan

Re: Adsbot (Google) access?

Post by asinshesq »

Thanks, that's what I figured on the phpbb end. But any thoughts on why adbots (and a number of other bots) are listed in the manage spiders area as accessing the site even though I have a robots.txt file saying they shouldn't do that? Also, so that I can avoid users being concerned about this, anything wrong with disactivating all bots in the manage spiders list? (If there's a site efficiency reason to leave them activated I can always tinker with view online to suppress listing of bots.)
User avatar
5hocK
Registered User
Posts: 3011
Joined: Wed Nov 23, 2011 7:00 pm
Location: England

Re: Adsbot (Google) access?

Post by 5hocK »

Alot of bad bots tend to ignore robots.txt so it's best to keep the bots and spiders list (possibly add to it too*) as you can atleast control their activity and as special registered users their visits only appear as one visit no matter what IP they use (like a normal user) and some bots have hundreds of IPs which is a pain - delete a bot and it's possible to suddenly have Xhundred extra guests online in a day.

* Spiders & bots to add to phpBB
A thread is pretty much a topic. Stop with the pedantics will ya.
User avatar
Oyabun1
Former Team Member
Posts: 23162
Joined: Sun May 17, 2009 1:05 pm
Location: Australia
Name: Bill

Re: Adsbot (Google) access?

Post by Oyabun1 »

Your robots.txt file should work. However, check the file access permissions for it. Sometimes it seems hosts change permissions on files for unexplained reasons.
                      Support Request Template
3.0.x: Knowledge Base Styles Support MOD Requests
3.1.x: Knowledge BaseStyles SupportExtension Requests
User avatar
Lumpy Burgertushie
Registered User
Posts: 68114
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Adsbot (Google) access?

Post by Lumpy Burgertushie »

also, the bots that are listed in the bots group are members so they see whatever you have set the permissions for them to be able to see.
I don't know if your robots.txt file will stop that or not.


robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
asinshesq
Registered User
Posts: 6266
Joined: Sun Feb 22, 2004 9:34 pm
Location: NYC
Name: Alan

Re: Adsbot (Google) access?

Post by asinshesq »

Thanks, all.

Bill, good thought but I checked and I see robots.txt has 644 permissions.

Robert, I thought indicating disallow in robots.txt would tell properly functioning and compliant bots like google to skip the site altogether, and I have never assigned the bots group any role or granted them any permissions on any of the forums so I would have guessed that they (like guests) cannot see anything on my site. And in fact if I'm google my sitename the google result reads as follows (as you would expect):
A description for this result is not available because of this site's robots.txt
Locked

Return to “[3.0.x] Support Forum”