Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA »

J_M wrote:a google bot "iphone"?

any thoughts on this one that popped up in my forum? ( I hope this hasn't already been discussed )

Google [Bot] IP: 66.249.73.116 » Whois
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (co
It's probably just somebody on the Google campus using their iPhone to connect through Google's WiFi network. Google actually has people that work there; they're not all bots. ;)

Did it actually show up as "Google [Bot]" on your board? If so, you must have added something to the Bots list yourself.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

Did it actually show up as "Google [Bot]" on your board? If so, you must have added something to the Bots list yourself.
I haven't messed with the Bots list ( QuickAccess > Spiders/Robots ), but there is a line item for Google[Bot]. What triggers phpBB to identify a visitor as a bot?

but.....

I do use this mobile theme; http://www.artodia.com/phpbb-styles/mobile-iphone/

could the mobile device detection be at fault?

http://www.artodia.com/phpbb-styles/mob ... detection/
Last edited by J_M on Tue Dec 04, 2012 1:25 pm, edited 1 time in total.
leschek
Registered User
Posts: 791
Joined: Tue Jul 18, 2006 12:49 pm
Contact:

Re: Spiders & bots to add to phpBB

Post by leschek »

I also saw it on my forum and I think it is Google Mobile Crawler, but it doesn't show whole user agent. More here
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

leschek

thanks! that makes sense.
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA »

J_M wrote:
Did it actually show up as "Google [Bot]" on your board? If so, you must have added something to the Bots list yourself.
I haven't messed with the Bots list ( QuickAccess > Spiders/Robots ), but there is a line item for Google[Bot]. What triggers phpBB to identify a visitor as a bot?
I'm still not clear if the actual text in the Who's Online list showed "Google [Bot]". After reading leschek's article, I'm thinking that it did and the Who's Online list truncated the long user agent.

As for how a bot is determined, if a Guest is seen (somebody who isn't logged on already or automatically), the Bots table is checked for a matching User Agent piece and/or IP address. If a match is found, a bot is shown; otherwise a Guest is shown.

In this case, I'm guessing that the User Agent was truncated for display, but the full version contained Googlebot-Mobile in it. Because Google [Bot] uses Googlebot for its User Agent match, the mobile bot was flagged as the normal bot.

If i add Googlebot-Mobile to manage_bots, that should cause all versions of the mobile bot to be flagged correctly (because longer agent strings are checked before shorter ones).

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.
J_M
Registered User
Posts: 269
Joined: Wed Jul 20, 2005 12:26 pm

Re: Spiders & bots to add to phpBB

Post by J_M »

Hi Steve

>I'm still not clear if the actual text in the Who's Online list showed "Google [Bot]".

Yes, I cut and paste what was on the who's online listing.

thanks
Schwpz
Registered User
Posts: 335
Joined: Wed May 07, 2003 1:33 pm
Location: Planet Zot
Contact:

Re: Spiders & bots to add to phpBB

Post by Schwpz »

New one:

Bot name: Mandalay [Bot]
Agent match: mandalay
Information on the bot: http://boston.lti.cs.cmu.edu/crawler/clueweb12pp/
..:: PlanetZot.com - Your ultimate source for animation! ^^
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Schwpz wrote:New one:

Bot name: Mandalay [Bot]
Agent match: mandalay
Information on the bot: http://boston.lti.cs.cmu.edu/crawler/clueweb12pp/
It is a new agent match for the existing Lemur bot: viewtopic.php?p=13097692#p13097692

/Marcus
Schwpz
Registered User
Posts: 335
Joined: Wed May 07, 2003 1:33 pm
Location: Planet Zot
Contact:

Re: Spiders & bots to add to phpBB

Post by Schwpz »

Oh, didn't knew.
Thanks!
..:: PlanetZot.com - Your ultimate source for animation! ^^
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: Wada [Bot]
Agent match: Wada
User agent string: Wada.vn-Images Vietnamese Search/0.2
Website: http://wada.vn/congty/en/about-wada.html

Bot name: IntegromeDB [Bot]
Agent match: integromedb
User agent string: http://www.integromedb.org/Crawler
Website: http://www.integromedb.org

Bot name: JS-Kit [Bot]
Agent match: JS-Kit
User agent string: JS-Kit URL Resolver, http://js-kit.com/
Website: http://js-kit.com/

[Additional info]
Bot name: Ning [Bot]
Agent match: Ning
User agent string: NING/1.0
Website: http://www.ning.com

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: Liferea [Bot]
Agent match: Liferea
User agent string: Liferea/1.6.4 (Linux; da_DK.utf8; http://liferea.sf.net/)
Website: http://liferea.sf.net/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: NetworkedBlogs [Bot]
Agent match: NetworkedBlogs
User agent string: NetworkedBlogs (+http://www.networkedblogs.com; X subscribers) AppEngine-Google; (+http://code.google.com/appengine; appid: s~networkedblogshr)
Website: http://www.networkedblogs.com/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: Zite [Bot]
Agent match: zite
User agent string: Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
Website: http://www.zite.com

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: Mail.ru [Bot]
Agent match: mail.ru
User agent string: Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Website: http://www.mail.ru

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel »

Bot name: 360sou [Bot]
Agent match: 360Spider
User agent string: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Website: http://www.360sou.com

Bot name: GG.pl [Bot]
Agent match: gg.pl
User agent string: GG PeekBot 1.0 (http://gg.pl)
Website: http://gg.pl

/Marcus
Post Reply

Return to “phpBB Discussion”