Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

hi Steve,
it's a pretty safe bet that they accessed that URL directly.
Would the following be a safe assumption - if the user agent string of a "guest" doesn't include a bot name, and it's wandering around the UCP or memberlist, it's an unfriendly bot?

thanks again,

j
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA »

J_M wrote:
it's a pretty safe bet that they accessed that URL directly.
Would the following be a safe assumption - if the user agent string of a "guest" doesn't include a bot name, and it's wandering around the UCP or memberlist, it's an unfriendly bot?
Probably, but with some exceptions. The ability to view the Member List is permission-based, so some sites might allow guests to view that. It could also be a normal guest (a live person) just trying to see what they can see, although I'd probably classify him as a potential hacker.

Is there something specific you want to do? (Remember, this topic is really about listing new bots to add to phpBB, not about bots in general.)

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

this topic is really about listing new bots to add to phpBB
That's part of what I'm confused about. Trying to identify the bots. There are some bots that list their names such as ezoom, google etc, but where would I find the names of the other bots when the name isn't listed in the user agent string?

if this is not on topic, please let me know.. no worries.
User avatar
Lumpy Burgertushie
Registered User
Posts: 68007
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Lumpy Burgertushie »

J_M wrote:hi Steve,
it's a pretty safe bet that they accessed that URL directly.
Would the following be a safe assumption - if the user agent string of a "guest" doesn't include a bot name, and it's wandering around the UCP or memberlist, it's an unfriendly bot?

thanks again,

j
if it is a guest then it is not wandering around in the UCP or memberlist because those things are normally not available to guests. since it is only seeing either an error message or a login box there is really nothing to worry about.


robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

>if it is a guest

My understanding is that any bot that is not in the ACP list is identified as a guest within the "who is online" page. So, if it's an unidentified bot, how do I determine what it is? for example does xRumer leave any tell tale signs that can be identified by phpBB?

>there is really nothing to worry about

My concern is that if someone has programmed a bot that only has Guest privileges, to first identify that it is a phpBB forum, and then to search for and locate the UCP/MCP, there must be a goal? Even if it is currently not visible to them.

At this point it's just annoyances, seeing 10 instances of 360spider on the forum doesn't make me feel comfortable. I have just started to play with blocking their IPs, but I would guess that they are much smarter than I am and this will be a futile game.
User avatar
Lumpy Burgertushie
Registered User
Posts: 68007
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Lumpy Burgertushie »

J_M wrote:>if it is a guest

My understanding is that any bot that is not in the ACP list is identified as a guest within the "who is online" page. So, if it's an unidentified bot, how do I determine what it is? for example does xRumer leave any tell tale signs that can be identified by phpBB?

>there is really nothing to worry about

My concern is that if someone has programmed a bot that only has Guest privileges, to first identify that it is a phpBB forum, and then to search for and locate the UCP/MCP, there must be a goal? Even if it is currently not visible to them.

At this point it's just annoyances, seeing 10 instances of 360spider on the forum doesn't make me feel comfortable. I have just started to play with blocking their IPs, but I would guess that they are much smarter than I am and this will be a futile game.
I think you would be surprised to find how many attempts are made daily to access all different files on your server, not just the phpbb ones, by all sorts of automated scripts looking for ways to get in.

I can visit your site as a guest. I can attempt to view the ucp by typing in the url to it. However, unless you have your permissions set wrong, I will never be able to get to it. However, I will show up as a guest viewing your ucp.

knowing that the bots are attempting to access your site will not help you, it will not keep them from doing it etc.

as you said, banning via IP is pretty much a waste of your time and effort. the bots change IPs at will, even normal people will get a new IP quite often and let's not forget proxies.

luck,
robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

luck,
yup. I'll need it : )

I'll do test of blocking via htaccess for a month and see what happens. I know that I will sacrifice certain regions of the world. I'm not happy to limit anyones access, but the site doesn't get any real activity from these areas compared to the volume of hits that I'm seeing.

regards

j
User avatar
Lumpy Burgertushie
Registered User
Posts: 68007
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Lumpy Burgertushie »

just curious, what do you care? it is not affecting your board in any way.
go look at your server access logs if you want' to worry about who and what is trying to access your site every day.


robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

look at your server access logs if you want' to worry
exactly!

1/4 of the traffic/data is from a country where we don't have a single member. My site has had some problems in the past with resource usage, I'm going to try this and see if it has any effect.

j
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA »

J_M wrote: My understanding is that any bot that is not in the ACP list is identified as a guest within the "who is online" page.
That is correct. Guests can be one of three "things" -- true guests, registered users who don't get logged in automatically and bots that aren't listed in the ACP. The last can be good bots (search bots, validation tools, etc.) or bad bots (E-mail scrapers, spam bots, etc.).
J_M wrote: So, if it's an unidentified bot, how do I determine what it is? for example does xRumer leave any tell tale signs that can be identified by phpBB?
The best way is the User Agent. Of course, spam bots don't generally want you know who they are, so most will try to use generic browser User Agents. (I say "most" because i do have some "bad" bots listed in manage_bots by User Agent, including at least one that's supposed to be Xrumer.)

If you can't rely on the User Agent, the only other thing is the IP address. However, while good bots may publish the IP addresses they may visit from, spam bots want to disguise that. To hide their slimy identities, they use proxies or, worse, zombie machines belonging to innocent users. So blocking their IP addresses (even if in the U.S. or wherever you are), may block a potentially valid user who got infected.

So there's a risk/reward trade-off in IP address blocking. My philosophy is to block as little as you can, so instead of blocking 192.168.1.*, maybe block 192.168.1.16-31.
J_M wrote: At this point it's just annoyances, seeing 10 instances of 360spider on the forum doesn't make me feel comfortable. I have just started to play with blocking their IPs, but I would guess that they are much smarter than I am and this will be a futile game.
As Robert said, I wouldn't worry about it unless it starts impacting your site. If your site seems to be running slowly or you hit a bandwidth cap, then do something. Otherwise, it's more or less the cost of doing business on the Web.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
User avatar
Dog Cow
Registered User
Posts: 2499
Joined: Fri Jan 28, 2005 12:14 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Dog Cow »

Pony99CA wrote: Spambots like Xrumer probably have ways to decide what type of board or blog a site is running and take appropriate action to access those pages directly.
Yeah, they probably read the Powered By line. :?
Moof!
Mac GUI Vault: Retro Apple II & Macintosh computing archive.
Inside Allerton bookMac GUIMac 512K Blog
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

Otherwise, it's more or less the cost of doing business on the Web.
I do realize that we are playing in a big world, but the tools to reduce some of the problems seem to be really easy to enable. And so I've settled on my test for March. The rationale:

The benefit would be to reduce some chances of a zero-day exploit or having to boot off some Xrumer type thing that learns how to read the minds of the people that create the QA anti-spam questions and successfully registers. And, reducing unwanted traffic, resource use, and scraping of the site.

All of this weighed against the extremely limited and most likely zero impact to the audience of this particular site.

I found that the tools to set this up with available IP lists and .htaccess are really easy. It's not a perfect solution but it should reduce some of the traffic. If it does work and the IP lists become outdated I'll just update them.
User avatar
Lumpy Burgertushie
Registered User
Posts: 68007
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Lumpy Burgertushie »

that is a nice exercise but is really just that.

trying to ban via IP is just about useless. the spammers change IPs at will.

you have no way of knowing if a visitor is valid or not based on IP alone.

having your site visited by "bots" is not a problem. every site in the world is visited by different "bots" every minute of every day. I am not sure what you think you are going to accomplish with this but hey, go for it.

luck,
robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

I am not sure what you think you are going to accomplish with this but hey, go for it.
agreed, I don't claim to know what I'm doing, but I do know more about this than I did last week. And, at the end of the month I'll know a little more.

Since I donate my time to run this site, if I can see a way to reduce the maintenance time.. that's a good thing. If it doesn't work out, there's nothing lost.
User avatar
Lumpy Burgertushie
Registered User
Posts: 68007
Joined: Mon May 02, 2005 3:11 am
Contact:

Re: Spiders & bots to add to phpBB

Post by Lumpy Burgertushie »

at risk of belaboring this, what maintenance time are you referring to?

unless you have no spam protection turned on and/or allowing guest posting, the fact that "bots" are visiting your board requires you to do nothing. most people don't even know they are being visited by all the bots/guests etc.

why do you think this requires you to even know about it, much less have to do something about it?

robert
I'm baaaaaccckkkk. still doing work on donation basis. PM your needs.

Premium phpBB 3.3 Styles by PlanetStyles.net

If nobody is in the forest, does a tree really fall?
Post Reply

Return to “phpBB Discussion”