Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

bidouille wrote: Wed Nov 27, 2019 10:34 am Hello,

This Bot list is still being maintained?

Ecosia search engine seems to be missing.
I don't find any clue on it.
Same. Can't find a user agent for it. Any ideas?
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Spiders & bots to add to phpBB

Post by 3Di »

Code: Select all

    Mozilla/5.0 (iPad; CPU OS 12_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) (Ecosia [email protected])
    Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia [email protected])
    Mozilla/5.0 (Linux; Android 8.0.0; SM-A520F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia [email protected])
https://udger.com/resources/ua-list

https://user-agents.net/browsers/ecosia
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

Whatever you do, do not add AppleWebKit to your ACP as a bot on it's own. I did this before and nobody couldn't log in. It was weird because the "Login" button completely disappeared from the forum and no one could log in. Strange! 🤔
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
User avatar
Brf
Support Team Member
Support Team Member
Posts: 53400
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: Spiders & bots to add to phpBB

Post by Brf »

"AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

Brf wrote: Fri Jul 31, 2020 7:46 pm "AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.
That's a bit of a worry. What exactly causes the "login" to disappear from just adding these strings? The reason I ask is because we have quite a few bots added to our forum and want to know if any of our current bot "strings" are affecting our site :?
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
Paul
Infrastructure Team Leader
Infrastructure Team Leader
Posts: 28619
Joined: Sat Dec 04, 2004 3:44 pm
Location: The netherlands.
Name: Paul Sohier
Contact:

Re: Spiders & bots to add to phpBB

Post by Paul »

It is not worrying, you are trying to add a browser, which should never be done.
User avatar
AmigoJack
Registered User
Posts: 6108
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack »

Forex Station wrote: Sat Aug 01, 2020 10:07 amWhat exactly causes the "login" to disappear from just adding these strings?
That's the answer, not the question. The very point of distinguishing bots from people is that the former should never (be able to) log in, only the latter. Otherwise bots would end up encountering the login again and again although it's obvious they have no credentials to enter.
Forex Station wrote: Sat Aug 01, 2020 10:07 amwant to know if any of our current bot "strings" are affecting our site
Again: that's the very point that those definitions have an effect. It's also the very point to make them distinctive, as precise as possible. If you messed with it (as in: not understood how to formulate matches) then remove your adds again.

To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)
  • "The problem is probably not my English but you do not want to understand correctly. ... We will not come anybody anyway, nevertheless, it's best to shit this." Affin, 2018-11-20
  • "But this shit is not here for you. You can follow with your. Maybe the question, instead, was for you, who know, so you shoved us how you are." axe70, 2020-10-10
  • "My reaction is not to everyone, especially to you." Raptiye, 2021-02-28
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

AmigoJack wrote: Sat Aug 01, 2020 12:34 pm
To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)
Thanks for taking the time to write that up mate, I now understand it a bit better.

If in doubt, would adding a / at the end of the agent string help prevent any issues with strings?

For example, if i've got Google Pagespeed bot, this bot's string is
Chrome-Lighthouse. Seeing as this string has a browser name in it, and we shouldn't be adding browsers, would adding the forward slash at the end stop issues etc.?
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
User avatar
AmigoJack
Registered User
Posts: 6108
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack »

The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is. If you think (Ecosia / will match anything then you understood it all wrong. If you think Chrome-Lighthouse as a text that needs to match messes with the internet browser that only matches Chrome/ then you still understood it all wrong. Copy the 3 examples to your favorite editor, then hit CTRL+F and try to find matches what everything comes to your mind:
  • Chrome alone will match all 3 lines
  • Chrome-Lighthouse will match none
  • Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia [email protected]) will match only 1, but...
  • (Ecosia will match all 3 lines.
I thought it was more than obvious how searching text parts in texts works... We neither have regular expressions here, nor wildcards, nor any other magic or voodoo - only plain text. For what to find.
  • "The problem is probably not my English but you do not want to understand correctly. ... We will not come anybody anyway, nevertheless, it's best to shit this." Affin, 2018-11-20
  • "But this shit is not here for you. You can follow with your. Maybe the question, instead, was for you, who know, so you shoved us how you are." axe70, 2020-10-10
  • "My reaction is not to everyone, especially to you." Raptiye, 2021-02-28
User avatar
Forex Station
Registered User
Posts: 177
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia
Contact:

Re: Spiders & bots to add to phpBB

Post by Forex Station »

AmigoJack wrote: Sat Aug 01, 2020 5:42 pm The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is.
Understood. Thank you for clearing that up, got it now 👌
Highly-customized PhpBB board voted as one of the most influential trading sites in the world: forex-station.com 💬
User avatar
dbrewood
Registered User
Posts: 446
Joined: Tue Feb 09, 2021 9:35 am
Name: Daron Brewood
Contact:

Re: Spiders & bots to add to phpBB

Post by dbrewood »

I've just removed the 'Add Bot' Extension as it is labeled abandoned. So what is the best way of adding known bots to the phpBB system?
User avatar
david63
Registered User
Posts: 20646
Joined: Thu Dec 19, 2002 8:08 am

Re: Spiders & bots to add to phpBB

Post by david63 »

dbrewood wrote: Fri Dec 24, 2021 12:35 pm I've just removed the 'Add Bot' Extension as it is labeled abandoned. So what is the best way of adding known bots to the phpBB system?
Unless you have a list of them the only way I know of is to add them manually.
David
Remember: You only know what you know and - you don't know what you don't know!

I now no longer support any of my extensions but they will start to become available here
User avatar
dbrewood
Registered User
Posts: 446
Joined: Tue Feb 09, 2021 9:35 am
Name: Daron Brewood
Contact:

Re: Spiders & bots to add to phpBB

Post by dbrewood »

I was wondering if this was still a viable system?

https://phpbb.hifikabin.me.uk/viewtopic.php?f=6&t=25
User avatar
david63
Registered User
Posts: 20646
Joined: Thu Dec 19, 2002 8:08 am

Re: Spiders & bots to add to phpBB

Post by david63 »

dbrewood wrote: Fri Dec 24, 2021 12:52 pm I was wondering if this was still a viable system?
That, if I remember correctly, is just a list of bots formatted in such a way that it will add the bot records - it will not keep automatically updating as more bots are found
David
Remember: You only know what you know and - you don't know what you don't know!

I now no longer support any of my extensions but they will start to become available here
User avatar
dbrewood
Registered User
Posts: 446
Joined: Tue Feb 09, 2021 9:35 am
Name: Daron Brewood
Contact:

Re: Spiders & bots to add to phpBB

Post by dbrewood »

david63 wrote: Fri Dec 24, 2021 12:54 pm That, if I remember correctly, is just a list of bots formatted in such a way that it will add the bot records - it will not keep automatically updating as more bots are found
Yep I agree with that, I'm just thinking it'd be a good start to adding bots to the system. I could install AddBot again to add the list back that was in there (prior to me removing it), but as it's no longer supported........
Post Reply

Return to “phpBB Discussion”