Page 3 of 60

Re: Spiders & bots to add to phpbb3

Posted: Fri Jun 06, 2008 8:34 am
by Marcus Wendel
One more:

Bot name: Keywen Encyclopedia [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Bot

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sun Jun 15, 2008 11:26 am
by AllGo
Bot name: Ilse [Bot]
Agent match: INGRID
Information on the bot: ???

"INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)"

Re: Spiders & bots to add to phpbb3

Posted: Sat Jun 28, 2008 10:06 am
by Marcus Wendel
One more:

Bot name: Facebook [Bot]
Agent match: facebookexternalhit
Information on the bot: http://www.facebook.com/externalhit_uatext.php

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sun Jun 29, 2008 7:18 am
by romans1423
Is there any distinction between bots, spiders, and crawlers? I notice that everything listed here is a "bot" (excepting a few bad bots), but on the first page of the thread crawlers & spiders were mentioned as if they were distinct from bots.

What's the difference?

Re: Spiders & bots to add to phpbb3

Posted: Mon Jun 30, 2008 8:10 am
by Eelke
There is none, they are all terms for the same thing: automated scripts that follow links on the web and "harvest" data as they go along. "Bots" is the more generic term of the three, it could actually mean any script that performs an automated task (such as trying to automatically register accounts and posting spam using those accounts).

Re: Spiders & bots to add to phpbb3

Posted: Mon Jun 30, 2008 4:20 pm
by romans1423
That's what I figured, but then I saw that phpBB 3's default bots list contains listings for things marked [bot] and [crawler], and the screen itself is titled "Spiders/Robots."

I would have just called them all bots, but because somebody did not, I'm left naturally wondering what the actual difference, if any, is.

Re: Spiders & bots to add to phpbb3

Posted: Mon Jul 14, 2008 3:19 pm
by MartectX
Just an idea: A script to insert all these into the database would be great! :)

Re: Spiders & bots to add to phpbb3

Posted: Mon Jul 14, 2008 6:52 pm
by Raimon
MartectX wrote:Just an idea: A script to insert all these into the database would be great! :)
Create a add_bots.php file , copy this to it:

Code: Select all

<?php
/**
* 
* Add_bots
* This script add multiply bots to your database
* $Id: add_bots.php raimon $
*
*/

set_time_limit(0);

define('IN_PHPBB', true);
$phpbb_root_path = './';
$phpEx = substr(strrchr(__FILE__, '.'), 1);
include($phpbb_root_path . 'common.'.$phpEx);
include($phpbb_root_path . 'includes/functions_user.'.$phpEx);


// Start session management
$user->session_begin();
$auth->acl($user->data);
$user->setup();

$bots = array(
	'Twiceler [Bot]'			    => array('Twiceler', ''),
	'Voila [Bot]'			        => array('VoilaBot', ''),
	'Omgili [Bot]'			        => array('omgilibot', ''),
	'Noxtrum [Bot]'			        => array('noxtrumbot', ''),
	'Spinn3r [Bot]'			        => array('Spinn3r', ''),
	'Furl [Bot]'			        => array('FurlBot', ''),
	'CommonCrawl [Bot]'			    => array('CCBot', ''),
	'Naver [Bot]'			        => array('Yeti', ''),
	'BDProtect [Bot]'			    => array('BPImageWalker', ''),
	'Snap Shots [Bot]'			    => array('Snapbot', ''),
	'Whitevector [Bot]'			    => array('Whitevector Crawler', ''),
	'Hatena Antenna [Bot]'		    => array('Hatena Antenna', ''),
	'Snap Shots Preview [Bot]'      => array('SnapPreviewBot', ''),
	'Ilse [Bot]'			        => array('IlseBot', ''),
	'ImageShack [Bot]'			    => array('ImageShack Image Fetcher', ''),
	'Entireweb [Bot]'			    => array('Speedy Spider', ''),
	'Yandex [Bot]'			        => array('Yandex', ''),
	'WebCorp [Bot]'			        => array('WebCorp', ''),
	'WebAlta [Bot]'			        => array('WebAlta', ''),
	'Powerset [Bot]'			    => array('zermelo', ''),
	'Boston Project [SpamBot]'	    => array('Boston Project', ''),
	'Startpagina [Bot]'			    => array('Startpagina', ''),
	'Heeii [Bot]'			        => array('Heeii', ''),
	'Wget [SpamBot]'			    => array('Wget', ''),
	'Yodao [Bot]'			        => array('YodaoBot', ''),
	'vBSEO [Bot]'			        => array('vBSEO', ''),
	'WiseGuys [Bot]'			    => array('Vagabondo', ''),
	'Searchme [Bot]'			    => array('Charlotte', ''),
	'Exalead [Bot]'			        => array('Exabot', ''),
	'Yahoo Search Marketing [Bot]'	=> array('YahooYSMcm', ''),
	'Daum [Bot]'			        => array('Daumoa', ''),
	'webcollage [Bot]'			    => array('webcollage', ''),
	'Babaloo [Bot]'			        => array('BabalooSpider', ''),
	'Keywen Encyclopedia [Bot]'		=> array('KeywenBot', ''),
	'Facebook [Bot]'			    => array('facebookexternalhit', ''),
	
	
);
	
$bot_ids = array();
user_get_id_name($bot_ids, array_keys($bots), USER_IGNORE);
// call request
add_bots($bots);
echo 'You are finished, you have added new bots to the bots list!';


/**
* Add the search bots into the database
* This code should be used in execute_last if the source database did not have bots
* If you are converting bots this function should not be called
* @todo We might want to look at sharing the bot list between the install code and this code for consistency
*/
function add_bots($bots)
{
	global $db, $config;

	$sql = 'SELECT group_id FROM ' . GROUPS_TABLE . " WHERE group_name = 'BOTS'";
	$result = $db->sql_query($sql);
	$group_id = (int) $db->sql_fetchfield('group_id', false, $result);
	$db->sql_freeresult($result);

	if (!$group_id)
	{
		add_default_groups();

		$sql = 'SELECT group_id FROM ' . GROUPS_TABLE . " WHERE group_name = 'BOTS'";
		$result = $db->sql_query($sql);
		$group_id = (int) $db->sql_fetchfield('group_id', false, $result);
		$db->sql_freeresult($result);

	}




	foreach ($bots as $bot_name => $bot_ary)
	{
		$user_row = array(
			'user_type'				=> USER_IGNORE,
			'group_id'				=> $group_id,
			'username'				=> $bot_name,
			'user_regdate'			=> time(),
			'user_password'			=> '',
			'user_colour'			=> '9E8DA7',
			'user_email'			=> '',
			'user_lang'				=> $config['default_lang'],
			'user_style'			=> $config['default_style'],
			'user_timezone'			=> $config['board_timezone'],
			'user_allow_massemail'	=> 0,
		);

		$user_id = user_add($user_row);

		if ($user_id)
		{
			$sql = 'INSERT INTO ' . BOTS_TABLE . ' ' . $db->sql_build_array('INSERT', array(
				'bot_active'	=> 1,
				'bot_name'		=> $bot_name,
				'user_id'		=> $user_id,
				'bot_agent'		=> $bot_ary[0],
				'bot_ip'		=> $bot_ary[1])
			);
			$db->sql_query($sql);
		}
	}
}

?>
save it, upload it to the root directory of your webhost ( where you find config.php ) after that call it with your web browser yoursite.com/phpBB3/add_bots.php
Don't forget before you begin make a back up of your database for the case something going wrong.
After you are done, deleted this file.

Re: Spiders & bots to add to phpbb3

Posted: Mon Jul 14, 2008 9:42 pm
by MartectX
Woah, thank you Raimon! :D

Re: Spiders & bots to add to phpbb3

Posted: Sun Jul 20, 2008 3:57 pm
by Marcus Wendel
Two new ones added today:

Bot name: DotNetDotCom.org [Bot]
Agent match: DotBot
Information on the bot: http://www.dotnetdotcom.org/#info

Bot name: MSN Mobile [Bot]
Agent match: MSMOBOT
Information on the bot: no official Microsoft page found

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sun Jul 20, 2008 8:41 pm
by Marcus Wendel
Marcus Wendel wrote:Bot name: Keywen Encyclopedia [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Bot
Corrected information on this bot:

Bot name: Keywen Encyclopedia Links [Bot]
Agent match: KeywenBot
Information on the bot: http://www.keywen.com/Encyclopedia/Links/


Two new ones:

Bot name: Keywen Encyclopedia [Bot]
Agent match: EasyDL
Information on the bot: http://www.keywen.com/Encyclopedia/Bot/

Bot name: Soso [Bot]
Agent match: Sosospider
Information on the bot: http://help.soso.com/webspider.htm (Chinese search engine)

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sun Jul 20, 2008 9:59 pm
by John P
82.99.30.13,82.99.30.15,82.99.30.16,82.99.30.22,82.99.30.31,82.99.30.35,82.99.30.49,82.99.30.73

Is a Muxanet bot but I can't save it because it doesn't recognize the host.

How come?

Re: Spiders & bots to add to phpbb3

Posted: Wed Jul 23, 2008 1:10 pm
by reptileguy
Hi Stef775,

I have blocked the entire range of 82.99.30.xx because it annoyed me. This bot uses multiple IPs within that range which makes it look like you have 10 visitors on your site.

Re: Spiders & bots to add to phpbb3

Posted: Wed Jul 23, 2008 3:39 pm
by reptileguy
By the way, MSN bot also uses multiple IP-addresses, most of which appear as guests. (65.55.109.xxx & 65.55.110.xxx)
I don't want to block this one. Is there a way to add all these IPs to one account?

Re: Spiders & bots to add to phpbb3

Posted: Wed Jul 23, 2008 3:44 pm
by John P
I don't want to block it, I wan't to assign this iprange to one bot.

But if I do the bot can't be saved, so how can I save it?