Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
User avatar
Binano
Registered User
Posts: 68
Joined: Thu Jun 25, 2015 9:24 am

Re: Spiders & bots to add to phpBB

Post by Binano »

User avatar
thunderchero
Registered User
Posts: 129
Joined: Sun Nov 03, 2013 10:16 pm

Re: Spiders & bots to add to phpBB

Post by thunderchero »

sorry to bring up old topic, but does this work on 3.2?

I also plan on adding "hifikabin" Block Bad Bots with .HTACCESS
https://phpbb.hifikabin.me.uk/viewtopic.php?f=6&t=110

thunderchero
User avatar
pikachuturkey
Registered User
Posts: 335
Joined: Wed Dec 20, 2006 10:34 pm
Location: Türkiye(Turkey)
Name: Rıza

Re: Spiders & bots to add to phpBB

Post by pikachuturkey »

thunderchero wrote: Mon Apr 24, 2017 3:51 am
sorry to bring up old topic, but does this work on 3.2?
Yes it works.
v12mike
Registered User
Posts: 584
Joined: Thu Jul 09, 2015 5:03 pm

Re: Spiders & bots to add to phpBB

Post by v12mike »

I don't know if this Bot list is being maintained, but I spotted 3 active bots that are not in it:

Name / Agent
megaindex [Crawler] / megaindex.com
Qwant [BOT] / Qwantify
British Library [Bot] / bl.uk_lddc_bot
User avatar
open-your-eyes
Registered User
Posts: 94
Joined: Mon Jul 31, 2017 8:20 am
Contact:

Re: Spiders & bots to add to phpBB

Post by open-your-eyes »

Hi gues!

I would very much apreciated if someone could point me out to the latest script for adding bots to 3.0. I want lots of them. :D
Thanks.
User avatar
HiFiKabin
Community Team Member
Community Team Member
Posts: 6669
Joined: Wed May 14, 2014 9:10 am
Location: Swearing at the PC, UK
Name: James
Contact:

Re: Spiders & bots to add to phpBB

Post by HiFiKabin »

As 3.0.x is no longer supported, I doubt anyone would want to make an up to date script. This post (with the script) was last updated in 2012 viewtopic.php?p=12119145#p12119145

You should also seriously consider upgrading to phpBB 3.2.x
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

Pony99CA wrote: Fri Feb 13, 2015 3:14 am
Puchahawa wrote:That would be a good Idea Steve thanks! Know of any other bad bots to add to the list?
Check the manage_bots code.
Puchahawa wrote: Would it be the right choice to change their default group to the bad bot group after they are added, or would it make more sense to remove them from the bot group?
You could certainly make the default group "Bad Bots". I don't know about removing them from the Bots group, though. I presume that wouldn't break their bot status, but I don't see a good reason to do it -- they are still bots, after all.

Steve
Steve is the man!

Thank you so much for this great tool. I love it so much. Could we get an updated manage_bots.php for 2018? Including DuckDuckGo etc? :)
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
User avatar
open-your-eyes
Registered User
Posts: 94
Joined: Mon Jul 31, 2017 8:20 am
Contact:

Re: Spiders & bots to add to phpBB

Post by open-your-eyes »

I am also interested in this.
User avatar
</Solidjeuh>
Registered User
Posts: 1788
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Spiders & bots to add to phpBB

Post by </Solidjeuh> »

Just search the bots you want on google and add them via ACP.
Like: https://duckduckgo.com/duckduckbot
User avatar
orynider
Translator
Posts: 270
Joined: Wed Nov 16, 2005 12:48 pm
Location: Arad, România
Name: Florin-Ciprian Bodin
Contact:

Re: Spiders & bots to add to phpBB

Post by orynider »

Here is an interesting source of bots https://github.com/monperrus/crawler-us ... gents.json

How about to add the e-mail address to the default 50 bots we have in the database?
User avatar
warmweer
Jr. Extension Validator
Posts: 11197
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium
Contact:

Re: Spiders & bots to add to phpBB

Post by warmweer »

orynider wrote: Thu Oct 25, 2018 5:22 pm How about to add the e-mail address to the default 50 bots we have in the database?
The problem here is that you'll need the 50 mailaddresses first 8-) .
Spelling is freeware, which means you can use it for free.
On the other hand, it is not open source, which means you cannot change it or publish it in a modified form.


Time flies like an arrow, but fruit flies like a banana.
bidouille
Registered User
Posts: 17
Joined: Fri Sep 15, 2006 3:27 pm
Location: France

Re: Spiders & bots to add to phpBB

Post by bidouille »

Hello,

This Bot list is still being maintained?

Ecosia search engine seems to be missing.
I don't find any clue on it.
User avatar
HiFiKabin
Community Team Member
Community Team Member
Posts: 6669
Joined: Wed May 14, 2014 9:10 am
Location: Swearing at the PC, UK
Name: James
Contact:

Re: Spiders & bots to add to phpBB

Post by HiFiKabin »

I am afraid it has not been updated in years. I only hosted it as a service for everyone. If anyone would like to actively support it, please contact me.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Spiders & bots to add to phpBB

Post by 3Di »

orynider wrote: Thu Oct 25, 2018 5:22 pm How about to add the e-mail address to the default 50 bots we have in the database?
Are you bound to mass-email them?
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
AmigoJack
Registered User
Posts: 6106
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack »

Many years ago I also thought about filling in the e-mail addresses: the profile field was there, so why not using it when the information is available? But since then there was not even one occurance where contacting the crawler support was needed (no, never mass-e-mailing, just looking up the address). And those crawlers I sent e-mailed to proved to never answer, let alone change their erratic software.
  • "The problem is probably not my English but you do not want to understand correctly. ... We will not come anybody anyway, nevertheless, it's best to shit this." Affin, 2018-11-20
  • "But this shit is not here for you. You can follow with your. Maybe the question, instead, was for you, who know, so you shoved us how you are." axe70, 2020-10-10
  • "My reaction is not to everyone, especially to you." Raptiye, 2021-02-28
Post Reply

Return to “phpBB Discussion”