Agent List for Bots/Spiders/Crawlers

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
Post Reply
User avatar
VeganFanatic
Registered User
Posts: 181
Joined: Mon Sep 08, 2008 8:27 pm
Location: Victoria, BC Canada
Contact:

Agent List for Bots/Spiders/Crawlers

Post by VeganFanatic »

I found this list while searching around for things with a range of search engines.

It has 302 active bots in it, so why cannot the phpBB group manage to update the database used by phpBB and modernize it?

Instead there are lots of dead bots that server no useful task.

ToonArmy
Former Team Member
Posts: 4608
Joined: Sat Mar 06, 2004 5:29 pm
Location: Worcestershire, UK
Name: Chris Smith
Contact:

Re: Agent List for Bots/Spiders/Crawlers

Post by ToonArmy »

Why does every board need 302 bots in the database?
Chris SmithGitHub

Drummer
Registered User
Posts: 199
Joined: Wed Jul 21, 2010 11:49 am

Re: Agent List for Bots/Spiders/Crawlers

Post by Drummer »

Actually, I have 9 of the default phpbb bots activated.
Many of the bots on this list are country-specific and most of them would never even try to visit your board.
It's not that hard to manually add the ones on who you have interest.

Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Agent List for Bots/Spiders/Crawlers

Post by Pony99CA »

VeganFanatic wrote:I found this list while searching around for things with a range of search engines.

It has 302 active bots in it, so why cannot the phpBB group manage to update the database used by phpBB and modernize it?

Instead there are lots of dead bots that server no useful task.
I've previously pointed you to my manage_bots script, which has more than 302 bots listed. The link in my signature is to a topic for discussing new bots to add to phpBB. If you want some of the ones from that list added, please feel free to post them in that topic using the format Marcus uses and we can add them.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
VeganFanatic
Registered User
Posts: 181
Joined: Mon Sep 08, 2008 8:27 pm
Location: Victoria, BC Canada
Contact:

Re: Agent List for Bots/Spiders/Crawlers

Post by VeganFanatic »

I was commenting on the main distribution and not a 3rd party solution.

I am suggesting that a new version bring new bots in so that forum users do not have to manually add them.

Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Agent List for Bots/Spiders/Crawlers

Post by Pony99CA »

VeganFanatic wrote:I was commenting on the main distribution and not a 3rd party solution.

I am suggesting that a new version bring new bots in so that forum users do not have to manually add them.
I understand, but my script could be used for that "new version", no? And the thread that I linked you to also had a link to another large list of bots.

Steve

P.S. I'm also at a loss to understand why this required a new topic. You started the Time to Clean up the Spiders/Bots topic to discuss basically the same thing, didn't you?
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

Post Reply

Return to “phpBB Discussion”