Google [Bot] Crawler Do's & Don'ts

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
.m.
Registered User
Posts: 364
Joined: Wed Nov 04, 2009 8:39 pm

Re: Adsense Ads Display Issues due to Permission

Post by .m. » Fri Mar 15, 2019 7:27 am

Adsense bot is expected to follow rules given in robots.txt file

for phpBB board restricted pages,
ordinary user permissions are required to access the pages

if bots have access to those restricted pages then
no need of supplying user login credentials to the Adsense bot

User avatar
Exclusive
Registered User
Posts: 327
Joined: Mon Jun 25, 2018 1:33 pm

Re: Adsense Ads Display Issues due to Permission

Post by Exclusive » Fri Mar 15, 2019 2:24 pm

.m. wrote:
Fri Mar 15, 2019 7:27 am
if bots have access to those restricted pages then
no need of supplying user login credentials to the Adsense bot
How do i grant the Adsense bot access to those restricted pages?

Below is what I set in my robots.txt

Code: Select all

User-agent: *
Crawl-delay: 1

User-agent: *
Disallow: /adm/
Disallow: /assets/
Disallow: /bin/
Disallow: /cache/
Disallow: /config/
Disallow: common.php
Disallow: /config/
Disallow: /contrib/
Disallow: /docs/
Disallow: /download/
Disallow: /ext/
Disallow: /includes/
Disallow: /language/
Disallow: /mobiquo/
Disallow: /phpbb/
Disallow: /skt/
Disallow: /store/
Disallow: /styles/
Disallow: /vendor/
Disallow: /app.php
Disallow: /common.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /exporttogroups.php
Disallow: /fail.php
Disallow: /faq.php
Disallow: /feed.php
Disallow: /index.php
Disallow: /install.txt
Disallow: /license.txt
Disallow: /mcp.php
Disallow: /memberlist.php
Disallow: /posting.php
Disallow: /report.php
Disallow: /search.php
Disallow: /success.php
Disallow: /ucp.php
Disallow: /viewonline.php
Disallow: /web.config

User-agent: NinjaBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Mediapaartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Adsbot-Google
Allow: /
 
User-agent: Googlebot-Mobile
Allow: /

User-agent: Yahoo-MMCrawler
Allow: /
 
User-agent: Yahoo-slurp
Allow: /

User-agent: slurp
Allow: /
 
User-agent: Msnbot
Allow: /

Sitemap: https://mydomain.com/sitemap.xml
Any help on this phpBB community??
Last edited by thecoalman on Fri Mar 15, 2019 3:32 pm, edited 1 time in total.
Reason: Merged with existing topic.
Use Search Button to get Instant Results | Follow Forum Rules
Avoid Spams in the forum to prevent banning the offending account. | Visit My Website!

Subscribe to my YouTube Channel!

Get The Best Web Hosting | Buy Cheap Domain Names

Post Reply

Return to “[3.2.x] Support Forum”

Who is online

Users browsing this forum: Google [Bot], MosesTM, richxrich, skatebord42er and 24 guests