Controlling Search Bot Crawling?

This forum is now closed as part of retiring phpBB2
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

This forum is now closed due to phpBB2.0 being retired.
Post Reply
neps
Registered User
Posts: 66
Joined: Thu Nov 21, 2002 12:38 pm

Controlling Search Bot Crawling?

Post by neps » Tue May 03, 2005 7:15 pm

I'm trying to figure something out and searching has only led to results of people wanting Google to hit them.

I don't have that problem, google hits me fine. Too fine in fact. It is my number one bandwidth each month, mostly hitting the viewtopic file. Just this month alone it's bots and msns have already totaled 1 gig in transfer, and my host only allows 15 gigs a month, so you can see where this is going, I'm always paying for extra bandwidth.

I don't want to turn google access off, cause I like the referring. I don't want to turn of Guest cause I like people being able to check out the forum before they decide to post.

But there has to be some sort of throttle so I can control their leeching, isn't there? Is it like a no follow or return in X amount of days type thing?


Thanks, and sorry if I missed the post, but I did search man.

Dagon
Registered User
Posts: 422
Joined: Tue Jun 03, 2003 8:59 pm

Post by Dagon » Tue May 03, 2005 8:02 pm

Well, you can use a robots.txt to limit it to only certain areas, that way it won't waste bandwidth in your user's profiles and member list and such.

neps
Registered User
Posts: 66
Joined: Thu Nov 21, 2002 12:38 pm

Post by neps » Wed May 04, 2005 12:47 am

hmm, I'm already using this, was hoping for something more. Does anyone use the following meta? Does it help?

<meta name="revisit-after" content="1 days" />

crappytheclown
Registered User
Posts: 106
Joined: Tue Mar 29, 2005 10:50 pm
Location: NYC
Contact:

Post by crappytheclown » Wed May 04, 2005 2:34 am

well i dont hav ean answer but can u give me some tips on how to get on google? does your host take care of it or did u do some stuff maually the way all the submmission site suggest???
Image

neps
Registered User
Posts: 66
Joined: Thu Nov 21, 2002 12:38 pm

Post by neps » Wed May 04, 2005 6:42 pm

If you do a forum search there is like a 40 page thread on that subject. Not sure what I've done anymore, I've used phpbb for a number of years now.


Still would be curious about my original question.

Thanks

Post Reply

Return to “[2.0.x] MOD Requests”