Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
User avatar
Forex Station
Registered User
Posts: 112
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Spiders & bots to add to phpBB

Post by Forex Station »

bidouille wrote:
Wed Nov 27, 2019 10:34 am
Hello,

This Bot list is still being maintained?

Ecosia search engine seems to be missing.
I don't find any clue on it.
Same. Can't find a user agent for it. Any ideas?
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
User avatar
3Di
Former Team Member
Posts: 16032
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: Spiders & bots to add to phpBB

Post by 3Di »

Code: Select all

    Mozilla/5.0 (iPad; CPU OS 12_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) (Ecosia ios@4.0.6.795)
    Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128)
    Mozilla/5.0 (Linux; Android 8.0.0; SM-A520F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128)
https://udger.com/resources/ua-list

https://user-agents.net/browsers/ecosia
Please PM me only to request paid works. Thx.
Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user
Extensions, Scripts, MOD porting, Update/Upgrades
Looking for a specific feature or alternative option? We will rock you! 🚀
User avatar
Forex Station
Registered User
Posts: 112
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Spiders & bots to add to phpBB

Post by Forex Station »

Whatever you do, do not add AppleWebKit to your ACP as a bot on it's own. I did this before and nobody couldn't log in. It was weird because the "Login" button completely disappeared from the forum and no one could log in. Strange! 🤔
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
User avatar
Brf
Support Team Member
Support Team Member
Posts: 52106
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: Spiders & bots to add to phpBB

Post by Brf »

"AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.
User avatar
Forex Station
Registered User
Posts: 112
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Spiders & bots to add to phpBB

Post by Forex Station »

Brf wrote:
Fri Jul 31, 2020 7:46 pm
"AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.
That's a bit of a worry. What exactly causes the "login" to disappear from just adding these strings? The reason I ask is because we have quite a few bots added to our forum and want to know if any of our current bot "strings" are affecting our site :?
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
Paul
Infrastructure Team Leader
Infrastructure Team Leader
Posts: 26812
Joined: Sat Dec 04, 2004 3:44 pm
Location: The netherlands.
Name: Paul Sohier
Contact:

Re: Spiders & bots to add to phpBB

Post by Paul »

It is not worrying, you are trying to add a browser, which should never be done.
Knock knock
Race condition
Who's there?

My BlogMy Photosmy phpBB Extensionscustom phpBB work & Development
User avatar
AmigoJack
Registered User
Posts: 5774
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack »

Forex Station wrote:
Sat Aug 01, 2020 10:07 am
What exactly causes the "login" to disappear from just adding these strings?
That's the answer, not the question. The very point of distinguishing bots from people is that the former should never (be able to) log in, only the latter. Otherwise bots would end up encountering the login again and again although it's obvious they have no credentials to enter.
Forex Station wrote:
Sat Aug 01, 2020 10:07 am
want to know if any of our current bot "strings" are affecting our site
Again: that's the very point that those definitions have an effect. It's also the very point to make them distinctive, as precise as possible. If you messed with it (as in: not understood how to formulate matches) then remove your adds again.

To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.
User avatar
Forex Station
Registered User
Posts: 112
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Spiders & bots to add to phpBB

Post by Forex Station »

AmigoJack wrote:
Sat Aug 01, 2020 12:34 pm

To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)
Thanks for taking the time to write that up mate, I now understand it a bit better.

If in doubt, would adding a / at the end of the agent string help prevent any issues with strings?

For example, if i've got Google Pagespeed bot, this bot's string is
Chrome-Lighthouse. Seeing as this string has a browser name in it, and we shouldn't be adding browsers, would adding the forward slash at the end stop issues etc.?
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
User avatar
AmigoJack
Registered User
Posts: 5774
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack »

The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is. If you think (Ecosia / will match anything then you understood it all wrong. If you think Chrome-Lighthouse as a text that needs to match messes with the internet browser that only matches Chrome/ then you still understood it all wrong. Copy the 3 examples to your favorite editor, then hit CTRL+F and try to find matches what everything comes to your mind:
  • Chrome alone will match all 3 lines
  • Chrome-Lighthouse will match none
  • Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128) will match only 1, but...
  • (Ecosia will match all 3 lines.
I thought it was more than obvious how searching text parts in texts works... We neither have regular expressions here, nor wildcards, nor any other magic or voodoo - only plain text. For what to find.
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.
User avatar
Forex Station
Registered User
Posts: 112
Joined: Thu Apr 06, 2017 2:26 pm
Location: Australia

Re: Spiders & bots to add to phpBB

Post by Forex Station »

AmigoJack wrote:
Sat Aug 01, 2020 5:42 pm
The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is.
Understood. Thank you for clearing that up, got it now 👌
Proudly powered by phpBB for almost 4 years now: forex-station.com :mrgreen:
Global Alexa rank: 40,500
Post Reply

Return to “phpBB Discussion”