Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy » Wed Jul 23, 2008 4:06 pm

Here is a topic on that bot. It appears to be malicious, so I decided to block it.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Wed Jul 23, 2008 5:19 pm

reptileguy wrote:By the way, MSN bot also uses multiple IP-addresses, most of which appear as guests. (65.55.109.xxx & 65.55.110.xxx)
I don't want to block this one. Is there a way to add all these IPs to one account?
If you use the Agent match field you don't have to think about the ip-addresses.

/Marcus

User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy » Wed Jul 23, 2008 5:31 pm

There is 'msnbot-media/' & 'msnbot/' in the agent match fields of the two MSN bots, but many IPs in the ranges that I mentioned are not recognized as MSN bots. What should I add to the agent match fields?

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Wed Jul 23, 2008 5:38 pm

reptileguy wrote:There is 'msnbot-media/' & 'msnbot/' in the agent match fields of the two MSN bots, but many IPs in the ranges that I mentioned are not recognized as MSN bots. What should I add to the agent match fields?
Check what the agent identified itself as in either the log files or the who's online.

/Marcus

User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy » Wed Jul 23, 2008 6:10 pm

All these IPs identify themselves as 'Microsoft Corp' in the whois-popup.
In the access logs I see 'http://search.live.com/results.aspx?' followed by some search query.
I'm not sure what to add to the agent match field :roll:

User avatar
John P
Registered User
Posts: 1232
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P » Wed Jul 23, 2008 8:38 pm

My "bot" acts the same, just like a regular browser IE 6 so I need to assign the IP range for it just like you ban a IP range.

Isn't it possible in phpbb3?
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign

candysmall
Registered User
Posts: 416
Joined: Sun Sep 12, 2004 4:58 pm
Location: Boston, MA

Re: Spiders & bots to add to phpbb3

Post by candysmall » Thu Jul 24, 2008 3:31 am

I'm getting the following in my logs but I don't know how to find the agents etc:

http://web.mac.com/lrmoorejr/geckobot/
Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
FunWebProducts


Also, what are all these?
http://user-agents.org/

T50
Registered User
Posts: 333
Joined: Mon May 07, 2007 11:51 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by T50 » Thu Jul 24, 2008 3:38 am

hmm seems like quite a list...

perhaps a few to add to my bot mod.

fyi my mod is in the queue and it automatically adds 109 bots..

don't know if any news ones are listed here. but you can get a advance copy of the mod at STG
http://startrekguide.com/community/view ... 127&t=7140

PS it will give an general error if the bot already exists in the db.. one of the few things i have to fix for version 2
Free Webhosting with option of getting paid plan <-- No Ads, No Pop-ups, Absolutely free/ or use promo code T50 to get a 10% discount on paid plan!
Make a few $$ in under a few minutes, easy to do

Webmaster of a Northeast Wisconsin Boy Scout Troop Website, http://bsat50.info

candysmall
Registered User
Posts: 416
Joined: Sun Sep 12, 2004 4:58 pm
Location: Boston, MA

Re: Spiders & bots to add to phpbb3

Post by candysmall » Thu Jul 24, 2008 3:58 am

T50 wrote:hmm seems like quite a list...

perhaps a few to add to my bot mod.

fyi my mod is in the queue and it automatically adds 109 bots..

don't know if any news ones are listed here. but you can get a advance copy of the mod at STG
http://startrekguide.com/community/view ... 127&t=7140

PS it will give an general error if the bot already exists in the db.. one of the few things i have to fix for version 2
I just used it and it worked GREAT! Thanks! How can we get an update when bots are added?

T50
Registered User
Posts: 333
Joined: Mon May 07, 2007 11:51 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by T50 » Thu Jul 24, 2008 4:03 am

it would be after version 1.0.0 before i add more.. since 1.0.0 would be the MODDB release version..
which should be hopefully added in a few weeks, but once it gets in I will start to get more things done on it..

unfornately I am leaving in 3 days for a week at camp and am in the midst of getting ready.. but once it comes out I plan to get it able to add bots when bots already exist and also use the mod version check..

Plus I hope to have it add a log to the admin log..
Free Webhosting with option of getting paid plan <-- No Ads, No Pop-ups, Absolutely free/ or use promo code T50 to get a 10% discount on paid plan!
Make a few $$ in under a few minutes, easy to do

Webmaster of a Northeast Wisconsin Boy Scout Troop Website, http://bsat50.info

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat Jul 26, 2008 10:09 am

Stef775 wrote:My "bot" acts the same, just like a regular browser IE 6 so I need to assign the IP range for it just like you ban a IP range.

Isn't it possible in phpbb3?
You can't add a range, but it should work if you add "82.99.30".

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat Jul 26, 2008 10:11 am

Two more:

Bot name: Visvo [Bot]
Agent match: VisBot
Information on the bot: http://www.visvo.com/webmasters.html

Bot name: Munax [Bot]
IP match: 82.99.30 (they annoyingly do not provide any agent name or any other info to id this bot)
Information on the bot: http://www.munax.com/crawlingfaq.htm

/Marcus

User avatar
John P
Registered User
Posts: 1232
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P » Sat Jul 26, 2008 10:27 am

Marcus thanks


That did the trick!!
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign

User avatar
angesetelfes
Registered User
Posts: 1
Joined: Sat Jul 26, 2008 10:59 am
Location: France - Reims - Champagne
Contact:

Re: Spiders & bots to add to phpbb3

Post by angesetelfes » Sat Jul 26, 2008 11:16 am

Thanks to all for this work ;)

User avatar
3Di
Registered User
Posts: 11826
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano - Frankfurt
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di » Sun Jul 27, 2008 11:27 pm

Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: No registered users and 17 guests

cron