Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Fri Jun 06, 2008 8:34 am

One more:

Bot name: Keywen Encyclopedia [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Bot

/Marcus

AllGo
Registered User
Posts: 12
Joined: Sun May 04, 2008 12:48 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by AllGo » Sun Jun 15, 2008 11:26 am

Bot name: Ilse [Bot]
Agent match: INGRID
Information on the bot: ???

"INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)"

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat Jun 28, 2008 10:06 am

One more:

Bot name: Facebook [Bot]
Agent match: facebookexternalhit
Information on the bot: http://www.facebook.com/externalhit_uatext.php

/Marcus

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Spiders & bots to add to phpbb3

Post by romans1423 » Sun Jun 29, 2008 7:18 am

Is there any distinction between bots, spiders, and crawlers? I notice that everything listed here is a "bot" (excepting a few bad bots), but on the first page of the thread crawlers & spiders were mentioned as if they were distinct from bots.

What's the difference?

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Spiders & bots to add to phpbb3

Post by Eelke » Mon Jun 30, 2008 8:10 am

There is none, they are all terms for the same thing: automated scripts that follow links on the web and "harvest" data as they go along. "Bots" is the more generic term of the three, it could actually mean any script that performs an automated task (such as trying to automatically register accounts and posting spam using those accounts).

romans1423
Registered User
Posts: 1552
Joined: Sat Nov 02, 2002 4:44 pm
Location: Connersville, IN
Name: Rick Beckman
Contact:

Re: Spiders & bots to add to phpbb3

Post by romans1423 » Mon Jun 30, 2008 4:20 pm

That's what I figured, but then I saw that phpBB 3's default bots list contains listings for things marked [bot] and [crawler], and the screen itself is titled "Spiders/Robots."

I would have just called them all bots, but because somebody did not, I'm left naturally wondering what the actual difference, if any, is.

User avatar
MartectX
Translator
Posts: 1324
Joined: Wed Dec 19, 2007 8:05 pm
Location: Marienplatz

Re: Spiders & bots to add to phpbb3

Post by MartectX » Mon Jul 14, 2008 3:19 pm

Just an idea: A script to insert all these into the database would be great! :)

User avatar
Raimon
Former Team Member
Posts: 12088
Joined: Tue May 30, 2006 5:31 pm
Location: Netherlands
Name: Raimon Meuldijk
Contact:

Re: Spiders & bots to add to phpbb3

Post by Raimon » Mon Jul 14, 2008 6:52 pm

MartectX wrote:Just an idea: A script to insert all these into the database would be great! :)
Create a add_bots.php file , copy this to it:

Code: Select all

<?php
/**
* 
* Add_bots
* This script add multiply bots to your database
* $Id: add_bots.php raimon $
*
*/

set_time_limit(0);

define('IN_PHPBB', true);
$phpbb_root_path = './';
$phpEx = substr(strrchr(__FILE__, '.'), 1);
include($phpbb_root_path . 'common.'.$phpEx);
include($phpbb_root_path . 'includes/functions_user.'.$phpEx);


// Start session management
$user->session_begin();
$auth->acl($user->data);
$user->setup();

$bots = array(
	'Twiceler [Bot]'			    => array('Twiceler', ''),
	'Voila [Bot]'			        => array('VoilaBot', ''),
	'Omgili [Bot]'			        => array('omgilibot', ''),
	'Noxtrum [Bot]'			        => array('noxtrumbot', ''),
	'Spinn3r [Bot]'			        => array('Spinn3r', ''),
	'Furl [Bot]'			        => array('FurlBot', ''),
	'CommonCrawl [Bot]'			    => array('CCBot', ''),
	'Naver [Bot]'			        => array('Yeti', ''),
	'BDProtect [Bot]'			    => array('BPImageWalker', ''),
	'Snap Shots [Bot]'			    => array('Snapbot', ''),
	'Whitevector [Bot]'			    => array('Whitevector Crawler', ''),
	'Hatena Antenna [Bot]'		    => array('Hatena Antenna', ''),
	'Snap Shots Preview [Bot]'      => array('SnapPreviewBot', ''),
	'Ilse [Bot]'			        => array('IlseBot', ''),
	'ImageShack [Bot]'			    => array('ImageShack Image Fetcher', ''),
	'Entireweb [Bot]'			    => array('Speedy Spider', ''),
	'Yandex [Bot]'			        => array('Yandex', ''),
	'WebCorp [Bot]'			        => array('WebCorp', ''),
	'WebAlta [Bot]'			        => array('WebAlta', ''),
	'Powerset [Bot]'			    => array('zermelo', ''),
	'Boston Project [SpamBot]'	    => array('Boston Project', ''),
	'Startpagina [Bot]'			    => array('Startpagina', ''),
	'Heeii [Bot]'			        => array('Heeii', ''),
	'Wget [SpamBot]'			    => array('Wget', ''),
	'Yodao [Bot]'			        => array('YodaoBot', ''),
	'vBSEO [Bot]'			        => array('vBSEO', ''),
	'WiseGuys [Bot]'			    => array('Vagabondo', ''),
	'Searchme [Bot]'			    => array('Charlotte', ''),
	'Exalead [Bot]'			        => array('Exabot', ''),
	'Yahoo Search Marketing [Bot]'	=> array('YahooYSMcm', ''),
	'Daum [Bot]'			        => array('Daumoa', ''),
	'webcollage [Bot]'			    => array('webcollage', ''),
	'Babaloo [Bot]'			        => array('BabalooSpider', ''),
	'Keywen Encyclopedia [Bot]'		=> array('KeywenBot', ''),
	'Facebook [Bot]'			    => array('facebookexternalhit', ''),
	
	
);
	
$bot_ids = array();
user_get_id_name($bot_ids, array_keys($bots), USER_IGNORE);
// call request
add_bots($bots);
echo 'You are finished, you have added new bots to the bots list!';


/**
* Add the search bots into the database
* This code should be used in execute_last if the source database did not have bots
* If you are converting bots this function should not be called
* @todo We might want to look at sharing the bot list between the install code and this code for consistency
*/
function add_bots($bots)
{
	global $db, $config;

	$sql = 'SELECT group_id FROM ' . GROUPS_TABLE . " WHERE group_name = 'BOTS'";
	$result = $db->sql_query($sql);
	$group_id = (int) $db->sql_fetchfield('group_id', false, $result);
	$db->sql_freeresult($result);

	if (!$group_id)
	{
		add_default_groups();

		$sql = 'SELECT group_id FROM ' . GROUPS_TABLE . " WHERE group_name = 'BOTS'";
		$result = $db->sql_query($sql);
		$group_id = (int) $db->sql_fetchfield('group_id', false, $result);
		$db->sql_freeresult($result);

	}




	foreach ($bots as $bot_name => $bot_ary)
	{
		$user_row = array(
			'user_type'				=> USER_IGNORE,
			'group_id'				=> $group_id,
			'username'				=> $bot_name,
			'user_regdate'			=> time(),
			'user_password'			=> '',
			'user_colour'			=> '9E8DA7',
			'user_email'			=> '',
			'user_lang'				=> $config['default_lang'],
			'user_style'			=> $config['default_style'],
			'user_timezone'			=> $config['board_timezone'],
			'user_allow_massemail'	=> 0,
		);

		$user_id = user_add($user_row);

		if ($user_id)
		{
			$sql = 'INSERT INTO ' . BOTS_TABLE . ' ' . $db->sql_build_array('INSERT', array(
				'bot_active'	=> 1,
				'bot_name'		=> $bot_name,
				'user_id'		=> $user_id,
				'bot_agent'		=> $bot_ary[0],
				'bot_ip'		=> $bot_ary[1])
			);
			$db->sql_query($sql);
		}
	}
}

?>
save it, upload it to the root directory of your webhost ( where you find config.php ) after that call it with your web browser yoursite.com/phpBB3/add_bots.php
Don't forget before you begin make a back up of your database for the case something going wrong.
After you are done, deleted this file.
Need phpBB installation, extenstions, Styles or integrate phpBB with you website?
Contact me for fair prices and good service!

User avatar
MartectX
Translator
Posts: 1324
Joined: Wed Dec 19, 2007 8:05 pm
Location: Marienplatz

Re: Spiders & bots to add to phpbb3

Post by MartectX » Mon Jul 14, 2008 9:42 pm

Woah, thank you Raimon! :D

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun Jul 20, 2008 3:57 pm

Two new ones added today:

Bot name: DotNetDotCom.org [Bot]
Agent match: DotBot
Information on the bot: http://www.dotnetdotcom.org/#info

Bot name: MSN Mobile [Bot]
Agent match: MSMOBOT
Information on the bot: no official Microsoft page found

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun Jul 20, 2008 8:41 pm

Marcus Wendel wrote:Bot name: Keywen Encyclopedia [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Bot
Corrected information on this bot:

Bot name: Keywen Encyclopedia Links [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Links/


Two new ones:

Bot name: Keywen Encyclopedia [Bot]
Agent match: EasyDL
Information on the bot: http://www.keywen.com/Encyclopedia/Bot/

Bot name: Soso [Bot]
Agent match: Sosospider
Information on the bot: http://help.soso.com/webspider.htm (Chinese search engine)

/Marcus

User avatar
John P
Registered User
Posts: 1237
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P » Sun Jul 20, 2008 9:59 pm

82.99.30.13,82.99.30.15,82.99.30.16,82.99.30.22,82.99.30.31,82.99.30.35,82.99.30.49,82.99.30.73

Is a Muxanet bot but I can't save it because it doesn't recognize the host.

How come?
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign

User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy » Wed Jul 23, 2008 1:10 pm

Hi Stef775,

I have blocked the entire range of 82.99.30.xx because it annoyed me. This bot uses multiple IPs within that range which makes it look like you have 10 visitors on your site.

User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy » Wed Jul 23, 2008 3:39 pm

By the way, MSN bot also uses multiple IP-addresses, most of which appear as guests. (65.55.109.xxx & 65.55.110.xxx)
I don't want to block this one. Is there a way to add all these IPs to one account?

User avatar
John P
Registered User
Posts: 1237
Joined: Mon Jan 21, 2008 3:55 pm
Location: Netherlands
Name: John
Contact:

Re: Spiders & bots to add to phpbb3

Post by John P » Wed Jul 23, 2008 3:44 pm

I don't want to block it, I wan't to assign this iprange to one bot.

But if I do the bot can't be saved, so how can I save it?
Image
Webhosting, Custom MODs, Technical management, MOD installation and Webdesign

Post Reply

Return to “phpBB Discussion”