Page 1 of 1

Controlling Search Bot Crawling?

Posted: Tue May 03, 2005 7:15 pm
by neps
I'm trying to figure something out and searching has only led to results of people wanting Google to hit them.

I don't have that problem, google hits me fine. Too fine in fact. It is my number one bandwidth each month, mostly hitting the viewtopic file. Just this month alone it's bots and msns have already totaled 1 gig in transfer, and my host only allows 15 gigs a month, so you can see where this is going, I'm always paying for extra bandwidth.

I don't want to turn google access off, cause I like the referring. I don't want to turn of Guest cause I like people being able to check out the forum before they decide to post.

But there has to be some sort of throttle so I can control their leeching, isn't there? Is it like a no follow or return in X amount of days type thing?

Thanks, and sorry if I missed the post, but I did search man.

Posted: Tue May 03, 2005 8:02 pm
by Dagon
Well, you can use a robots.txt to limit it to only certain areas, that way it won't waste bandwidth in your user's profiles and member list and such.

Posted: Wed May 04, 2005 12:47 am
by neps
hmm, I'm already using this, was hoping for something more. Does anyone use the following meta? Does it help?

<meta name="revisit-after" content="1 days" />

Posted: Wed May 04, 2005 2:34 am
by crappytheclown
well i dont hav ean answer but can u give me some tips on how to get on google? does your host take care of it or did u do some stuff maually the way all the submmission site suggest???

Posted: Wed May 04, 2005 6:42 pm
by neps
If you do a forum search there is like a 40 page thread on that subject. Not sure what I've done anymore, I've used phpbb for a number of years now.

Still would be curious about my original question.