Blocking Bad Bots

Looking for a MOD? Have a MOD request? Post here for help. (Note: This forum is community supported; phpBB does not have official MOD authors)
Ideas Centre
Locked
romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Blocking Bad Bots

Post by romans1423 »

As an extension to the Bots/Spiders administration, would it be possible to code something up that would allow added bots to be marked as "bad," redirecting them to 127.0.0.1 (i.e., themselves) or elsewhere (like a ProjectHoneyPot.com honey pot page)? It seems like this would be easy enough to do, and because the bots would be unable to access the site, resources would be saved, allowing the board to run that much better for legit users & bots.

So, to clarify:

When adding/editing a bot, there would be two extra options:

1) Mark this bot as "Bad."
2) Redirect this bot? With the options of either 127.0.0.1 or a configurable URL.

#2 would only take effect if #1 is marked yes, of course.

I think this would be a great tool to fight against spam or even content theft where IP addresses or User-Agents (like Bitacle, the rather infamous blog-content scraper) can be identified.

User avatar
damnian
Registered User
Posts: 801
Joined: Sat Nov 19, 2005 4:43 pm

Re: Blocking Bad Bots

Post by damnian »

Since bots are actually users, you may simply ban them by both username and IP from user admin... A shortcut from bot management should make it the perfect solution.
I no longer develop for phpBB. Check out my .NET blog!

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

I prefer not to ban by IP, though I suppose I could. But would adding a bot in the Bot Panel and then banning said username work?

Ideally, though, I still think it'd be neat to be able to control where the bot is redirected to, in order to aid such as projecthoneypot.com or the like.

User avatar
Highway of Life
Former Team Member
Posts: 6048
Joined: Wed Feb 02, 2005 5:41 pm
Location: Seattle, WA
Name: David Lewis
Contact:

Re: Blocking Bad Bots

Post by Highway of Life »

For bad bots, I usually just take the username -- which is the same as the bot name -- and add it to the username ban list.
You don’t need to ban by IP, since you could’ve already specified the IP when creating the BOT.
I’ve tested this method and it works quite well. :) -- At least this is the simplest default solution.

There is a "Disable bot" feature, but I don’t know what it does... you might try testing that, as it may achieve the desired effect as well.
The phpBB Weekly Podcast - Discussing the developments of phpBB4 and beyond.

New to phpBB3? Want to learn about programing?
Visit phpBB Academy at StarTrekGuide to learn how.

custmguru®
Registered User
Posts: 233
Joined: Wed Apr 10, 2002 6:06 pm
Location: Somewhere, Over the rainbow
Contact:

Re: Blocking Bad Bots

Post by custmguru® »

Highway of Life wrote:For bad bots, I usually just take the username -- which is the same as the bot name -- and add it to the username ban list.
You don’t need to ban by IP, since you could’ve already specified the IP when creating the BOT.
I’ve tested this method and it works quite well. :) -- At least this is the simplest default solution.

There is a "Disable bot" feature, but I don’t know what it does... you might try testing that, as it may achieve the desired effect as well.
the disable bot feature most likely will disable all bots good and bad.

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

But does it really? Or does it simply stop showing them as a "registered user" via sessions tracking, allowing them to still browse as a guest instead...

User avatar
Highway of Life
Former Team Member
Posts: 6048
Joined: Wed Feb 02, 2005 5:41 pm
Location: Seattle, WA
Name: David Lewis
Contact:

Re: Blocking Bad Bots

Post by Highway of Life »

custmguru® wrote:the disable bot feature most likely will disable all bots good and bad.
Actually, I was thinking of the individual bot setting: Bot Active: Yes/No
The phpBB Weekly Podcast - Discussing the developments of phpBB4 and beyond.

New to phpBB3? Want to learn about programing?
Visit phpBB Academy at StarTrekGuide to learn how.

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

"Disable bot" and "Bot active" settings are the same -- the terminology differs from the Spiders/Robots listing page and the individual Bot edit page.

Disabling or setting as inactive a bot will not prevent it from browsing; rather, it will simply cause it to show up as a regular guest.

Short of banning the usernames of a bot, there's no easy way to actually block bad bots (or, more preferably, redirecting them elsewhere).

And unfortunately, banning bots results in a Server Error due to incomplete script headers, so I can't even do that currently.

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

Blocking bad bots by banning their bot name does not work. It simply "logs them out" -- they are still free to browse the board. Indeed, banning anything (human or bot) by username is pointless if you have guest browsing and/or posting enabled. Banning by IP is more effective, but it still isn't foolproof (bad bots are more likely to have dynamic or at least various IPs upon which they browse than they are to have multiple user-agents).

So I would still like to see a MOD to extend the robots/spiders management to be able to set some bots as "bad bots" or "malicious" or whatever. Flagged bots wouldn't be "logged out" as they are with the banning utility; rather, they could either be fed a 403 forbidden error or 301 redirected elsewhere (admin-configurable, preferably).

coppro
Registered User
Posts: 84
Joined: Wed May 16, 2007 3:10 am

Re: Blocking Bad Bots

Post by coppro »

Right, but you can take each bot user by the hand (or the whole lot) and give them 'never' in every permissions category. They won't make it far.

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

coppro wrote:Right, but you can take each bot user by the hand (or the whole lot) and give them 'never' in every permissions category. They won't make it far.
Still isn't what my request is. I know phpBB has ways to control bots; my MOD request is for the bots/spiders facility to be extended to allow either total blockage (403 forbidden errors) or silent redirection of bots. I want to be able to force certain bots to honey pot pages created as part of ProjectHoneyPot.com.

User avatar
damnian
Registered User
Posts: 801
Joined: Sat Nov 19, 2005 4:43 pm

Re: Blocking Bad Bots

Post by damnian »

Hmm... interesting. However, chances are a bad bot won't handle a redirect. Maybe a standard ban with a link will do?
I no longer develop for phpBB. Check out my .NET blog!

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

damnian wrote:Hmm... interesting. However, chances are a bad bot won't handle a redirect. Maybe a standard ban with a link will do?
I've been redirecting them from my blog to a honey pot page for a while now, and my stats on Project Honey Pot have rocketed (they track honey pot page views) since I've done so. Redirects seem to work just fine for me. :)

User avatar
damnian
Registered User
Posts: 801
Joined: Sat Nov 19, 2005 4:43 pm

Re: Blocking Bad Bots

Post by damnian »

Maybe I'll cook up something. Should http:BL-blacklisted bots be redirected as well? ;-)
I no longer develop for phpBB. Check out my .NET blog!

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Blocking Bad Bots

Post by romans1423 »

damnian wrote:Maybe I'll cook up something. Should http:BL-blacklisted bots be redirected as well? ;-)
Well, if they're identified by http:BL as bad bots, why waste their visit? Definitely allow redirecting elsewhere -- a honey pot or QuickLink from Project Honey Pot, or even a page filled with nothing but junk emails, or whatever the user wants. :)

Locked

Return to “[3.0.x] MOD Requests”