Google [Bot] Crawler Do's & Don'ts

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Google [Bot] Crawler Do's & Don'ts

Post by Exclusive » Tue Mar 12, 2019 5:58 pm

Please is the phpBB Google Bot responsible for making google ads show or not showing in some specific pages? Or they are content crawlers only??
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 50413
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by stevemaury » Tue Mar 12, 2019 6:04 pm

Perhaps ask Google?
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)

User avatar
</Solidjeuh>
Registered User
Posts: 1547
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by </Solidjeuh> » Tue Mar 12, 2019 6:10 pm

It's the "Google Adsense [Bot]" , but this has nothing to do with phpBB.
Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Re: Google [Bot] Crawler Do's & Don'ts

Post by Exclusive » Tue Mar 12, 2019 6:24 pm

stevemaury wrote:
Tue Mar 12, 2019 6:04 pm
Perhaps ask Google?
Ask Google?? Okay thank you sir!
Last edited by Exclusive on Fri Mar 15, 2019 6:55 am, edited 2 times in total.
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 3162
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by thecoalman » Tue Mar 12, 2019 10:15 pm

</Solidjeuh> wrote:
Tue Mar 12, 2019 6:10 pm
It's the "Google Adsense [Bot]" , but this has nothing to do with phpBB.
It does in some sense, there is a reason Google has a separate bot. For example suppose you have chit chat forum you do not want indexed and you deny it to the bots group. You can create another group with the adsense bot or similar bots and grant them access. This will allow the ads bot to index it but keep those pages out of Google search results.

As another example I have a forum you need to be registered and then need to join a custom group with open access. I allow the adsense bot to index that forum. One thing to be aware of when doing this is someone could spoof the adsense user agent to get access so you do not want to do this with private forums with limited access.

That issue can likely be avoided too. I haven't tried it because I haven't had reasons but in your Adsense account you can enter login data, login URL etc. Not sure if phpBB allows you set password for a bot or allows it to login but if it does you should be able to let a bot like the adsense bot index content behind a login/limited access forum.

User avatar
Lumpy Burgertushie
Registered User
Posts: 66242
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by Lumpy Burgertushie » Tue Mar 12, 2019 10:49 pm

Exclusive wrote:
Tue Mar 12, 2019 5:58 pm
Please is the phpBB Google Bot responsible for making google ads show or not showing in some specific pages? Or they are content crawlers only??
ask google was the correct answer to your question.

how would we know what the google bot is doing any better than google does.


robert
I am available for custom work on a donation basis. Please send me a PM with your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

OK, so what's the speed of dark?

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Re: Google [Bot] Crawler Do's & Don'ts

Post by Exclusive » Wed Mar 13, 2019 5:23 am

thecoalman wrote:
Tue Mar 12, 2019 10:15 pm
but in your Adsense account you can enter login data, login URL etc. Not sure if phpBB allows you set password for a bot or allows it to login but if it does you should be able to let a bot like the adsense bot index content behind a login/limited access forum.
God Bless you sir, you're really on point... This was the main reason I oped this topic!

Some of my pages don't show my Adsense ad, mostly those pages that require login pages like:(/memberlist.php$) (but before those pages displayed ad, but I don't know what happened)... So I wonder which username an password the default (in-built) phpBB Google [Bot] uses to login so I could provide them in my Adsense Account? But I was asked to "ask Google" How does Google relate with phpBB?
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

User avatar
Lumpy Burgertushie
Registered User
Posts: 66242
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by Lumpy Burgertushie » Wed Mar 13, 2019 6:09 am

you asked about how the google bot works. we don't know. google knows. how phpbb handles google bot and other bots is another thing but that is not what you asked.

as for your adsense ads showing up on login pages etc. they are not supposed to. that is against google's terms of service. you can't put adsense ads on anything other than content type pages.

so, for phpbb that means only on the index_body.html, the viewforum_body.html and the viewtopic_body.html files

the bots are all in the bots group. they have their own set of permissions so that they can see the forums and topics. they do not "login" as such they are just allowed to view the code of the forums and topics/posts.

robert
I am available for custom work on a donation basis. Please send me a PM with your needs.

Premium phpBB 3.2 Styles by PlanetStyles.net

OK, so what's the speed of dark?

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Re: Google [Bot] Crawler Do's & Don'ts

Post by Exclusive » Wed Mar 13, 2019 6:55 am

Lumpy Burgertushie wrote:
Wed Mar 13, 2019 6:09 am
you asked about how the google bot works. we don't know. google knows. how phpbb handles google bot and other bots is another thing but that is not what you asked.

as for your adsense ads showing up on login pages etc. they are not supposed to. that is against google's terms of service. you can't put adsense ads on anything other than content type pages.

so, for phpbb that means only on the index_body.html, the viewforum_body.html and the viewtopic_body.html files

the bots are all in the bots group. they have their own set of permissions so that they can see the forums and topics. they do not "login" as such they are just allowed to view the code of the forums and topics/posts.

robert
Well said! Thank you.
Lumpy Burgertushie wrote:
Wed Mar 13, 2019 6:09 am
as for your adsense ads showing up on login pages etc. they are not supposed to. that is against google's terms of service. you can't put adsense ads on anything other than content type pages.
https://support.google.com/adsense/answer/161351
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

User avatar
KevC
Support Team Member
Support Team Member
Posts: 69038
Joined: Fri Jun 04, 2004 10:44 am
Location: Oxford, UK
Contact:

Re: Google [Bot] Crawler Do's & Don'ts

Post by KevC » Wed Mar 13, 2019 9:18 am

Exclusive wrote:
Wed Mar 13, 2019 6:55 am
Lumpy Burgertushie wrote: ↑
Wed Mar 13, 2019 6:09 am
as for your adsense ads showing up on login pages etc. they are not supposed to. that is against google's terms of service. you can't put adsense ads on anything other than content type pages.
https://support.google.com/adsense/answer/161351
That's not the answer to the statement Lumpy made.
That's telling you that you can put them behind a login. Lumpy is talking about having them ON the login page. That is a non-content page type.

Google ads, search boxes or search results may not be:
Placed on any non-content-based page. (Does not apply to AdSense for Search or mobile AdSense for Search.)

https://support.google.com/adsense/answ ... 2?hl=en-GB
-:|:- Support Request Template -:|:-
Image
Cheap UK Hosting
"In the land of the blind the little green bloke with no pupils is king - init!"

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Adsense Ads Display Issues due to Permission

Post by Exclusive » Fri Mar 15, 2019 6:16 am

Please, I have been experiencing issues with adsense ads in my forum. Things were perfectly working before, not until my hosting account auto updated my board version, After restoring it back to the previous version it was, Now I no longer understand my board.

Most of the pages that don't display ads are those pages the requires some extra permission/login to access them (eg: memberlist.php | but memberlist.php?mode=contactadmin shows ads). Before they do display ads, I just don't know what happened the don't anymore.... :x It only shows a blank space which indicates that an ad is suppose to display there.

Please what permission has altered this, how can I fix this? Please, I need help!
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

.m.
Registered User
Posts: 438
Joined: Wed Nov 04, 2009 8:39 pm

Re: Adsense Ads Display Issues due to Permission

Post by .m. » Fri Mar 15, 2019 6:49 am

looks like adsense bot requires login credentials for such restricted pages.

this might help --> https://support.google.com/adsense/answer/161351

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Re: Adsense Ads Display Issues due to Permission

Post by Exclusive » Fri Mar 15, 2019 6:53 am

.m. wrote:
Fri Mar 15, 2019 6:49 am
looks like adsense bot requires login credentials for such restricted pages.

this might help --> https://support.google.com/adsense/answer/161351
I came across this link before now, but I'm wandering which login credential will i provide to them?? That was why I asked this question here: viewtopic.php?f=556&t=2505806
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

.m.
Registered User
Posts: 438
Joined: Wed Nov 04, 2009 8:39 pm

Re: Adsense Ads Display Issues due to Permission

Post by .m. » Fri Mar 15, 2019 7:00 am

you can create a regular user account and provide its login credentials at that AdSense setting

User avatar
Exclusive
Registered User
Posts: 372
Joined: Mon Jun 25, 2018 1:33 pm

Re: Adsense Ads Display Issues due to Permission

Post by Exclusive » Fri Mar 15, 2019 7:05 am

.m. wrote:
Fri Mar 15, 2019 7:00 am
you can create a regular user account and provide its login credentials at that AdSense setting
Okay.... Just a registered user? No other extra permissions right?

But what about the phpBB Google Adsense [Bot] ? what username and password do the use?
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | ⟿ Visit My Forum! ⬳

Subscribe to my YouTube Channel!

Get The Best Web Hosting | ✔ Buy Cheap Domain Names

Post Reply

Return to “[3.2.x] Support Forum”