Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy »

Here is a topic on that bot. It appears to be malicious, so I decided to block it.
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

reptileguy wrote:By the way, MSN bot also uses multiple IP-addresses, most of which appear as guests. (65.55.109.xxx & 65.55.110.xxx)
I don't want to block this one. Is there a way to add all these IPs to one account?
If you use the Agent match field you don't have to think about the ip-addresses.

/Marcus
User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy »

There is 'msnbot-media/' & 'msnbot/' in the agent match fields of the two MSN bots, but many IPs in the ranges that I mentioned are not recognized as MSN bots. What should I add to the agent match fields?
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

reptileguy wrote:There is 'msnbot-media/' & 'msnbot/' in the agent match fields of the two MSN bots, but many IPs in the ranges that I mentioned are not recognized as MSN bots. What should I add to the agent match fields?
Check what the agent identified itself as in either the log files or the who's online.

/Marcus
User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy »

All these IPs identify themselves as 'Microsoft Corp' in the whois-popup.
In the access logs I see 'http://search.live.com/results.aspx?' followed by some search query.
I'm not sure what to add to the agent match field :roll:
User avatar
John P
Registered User
Posts: 1237
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P »

My "bot" acts the same, just like a regular browser IE 6 so I need to assign the IP range for it just like you ban a IP range.

Isn't it possible in phpbb3?
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign
candysmall
Registered User
Posts: 416
Joined: Sun Sep 12, 2004 4:58 pm
Location: Boston, MA

Re: Spiders & bots to add to phpbb3

Post by candysmall »

I'm getting the following in my logs but I don't know how to find the agents etc:

http://web.mac.com/lrmoorejr/geckobot/
Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
FunWebProducts


Also, what are all these?
http://user-agents.org/
T50
Registered User
Posts: 333
Joined: Mon May 07, 2007 11:51 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by T50 »

hmm seems like quite a list...

perhaps a few to add to my bot mod.

fyi my mod is in the queue and it automatically adds 109 bots..

don't know if any news ones are listed here. but you can get a advance copy of the mod at STG
http://startrekguide.com/community/view ... 127&t=7140

PS it will give an general error if the bot already exists in the db.. one of the few things i have to fix for version 2
Free Webhosting with option of getting paid plan <-- No Ads, No Pop-ups, Absolutely free/ or use promo code T50 to get a 10% discount on paid plan!
Make a few $$ in under a few minutes, easy to do

Webmaster of a Northeast Wisconsin Boy Scout Troop Website, http://bsat50.info
candysmall
Registered User
Posts: 416
Joined: Sun Sep 12, 2004 4:58 pm
Location: Boston, MA

Re: Spiders & bots to add to phpbb3

Post by candysmall »

T50 wrote:hmm seems like quite a list...

perhaps a few to add to my bot mod.

fyi my mod is in the queue and it automatically adds 109 bots..

don't know if any news ones are listed here. but you can get a advance copy of the mod at STG
http://startrekguide.com/community/view ... 127&t=7140

PS it will give an general error if the bot already exists in the db.. one of the few things i have to fix for version 2
I just used it and it worked GREAT! Thanks! How can we get an update when bots are added?
T50
Registered User
Posts: 333
Joined: Mon May 07, 2007 11:51 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by T50 »

it would be after version 1.0.0 before i add more.. since 1.0.0 would be the MODDB release version..
which should be hopefully added in a few weeks, but once it gets in I will start to get more things done on it..

unfornately I am leaving in 3 days for a week at camp and am in the midst of getting ready.. but once it comes out I plan to get it able to add bots when bots already exist and also use the mod version check..

Plus I hope to have it add a log to the admin log..
Free Webhosting with option of getting paid plan <-- No Ads, No Pop-ups, Absolutely free/ or use promo code T50 to get a 10% discount on paid plan!
Make a few $$ in under a few minutes, easy to do

Webmaster of a Northeast Wisconsin Boy Scout Troop Website, http://bsat50.info
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Stef775 wrote:My "bot" acts the same, just like a regular browser IE 6 so I need to assign the IP range for it just like you ban a IP range.

Isn't it possible in phpbb3?
You can't add a range, but it should work if you add "82.99.30".

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two more:

Bot name: Visvo [Bot]
Agent match: VisBot
Information on the bot: http://www.visvo.com/webmasters.html

Bot name: Munax [Bot]
IP match: 82.99.30 (they annoyingly do not provide any agent name or any other info to id this bot)
Information on the bot: http://www.munax.com/crawlingfaq.htm

/Marcus
User avatar
John P
Registered User
Posts: 1237
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P »

Marcus thanks


That did the trick!!
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign
angesetelfes
Registered User
Posts: 1
Joined: Sat Jul 26, 2008 10:59 am
Location: France - Reims - Champagne
Contact:

Re: Spiders & bots to add to phpbb3

Post by angesetelfes »

Thanks to all for this work ;)
User avatar
3Di
Former Team Member
Posts: 16174
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
Post Reply

Return to “phpBB Discussion”