Page 61 of 61

Re: Spiders & bots to add to phpBB

Posted: Fri Nov 29, 2019 6:36 am
by Forex Station
bidouille wrote:
Wed Nov 27, 2019 10:34 am
Hello,

This Bot list is still being maintained?

Ecosia search engine seems to be missing.
I don't find any clue on it.
Same. Can't find a user agent for it. Any ideas?

Re: Spiders & bots to add to phpBB

Posted: Fri Nov 29, 2019 7:09 am
by 3Di

Code: Select all

    Mozilla/5.0 (iPad; CPU OS 12_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) (Ecosia ios@4.0.6.795)
    Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128)
    Mozilla/5.0 (Linux; Android 8.0.0; SM-A520F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128)
https://udger.com/resources/ua-list

https://user-agents.net/browsers/ecosia

Re: Spiders & bots to add to phpBB

Posted: Fri Jul 31, 2020 7:41 pm
by Forex Station
Whatever you do, do not add AppleWebKit to your ACP as a bot on it's own. I did this before and nobody couldn't log in. It was weird because the "Login" button completely disappeared from the forum and no one could log in. Strange! 🤔

Re: Spiders & bots to add to phpBB

Posted: Fri Jul 31, 2020 7:46 pm
by Brf
"AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 10:07 am
by Forex Station
Brf wrote:
Fri Jul 31, 2020 7:46 pm
"AppleWebKit" is part of every agent string. The same thing would happen if you added "Mozilla" or "Gecko" or any of the other common strings.
That's a bit of a worry. What exactly causes the "login" to disappear from just adding these strings? The reason I ask is because we have quite a few bots added to our forum and want to know if any of our current bot "strings" are affecting our site :?

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 10:14 am
by Paul
It is not worrying, you are trying to add a browser, which should never be done.

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 12:34 pm
by AmigoJack
Forex Station wrote:
Sat Aug 01, 2020 10:07 am
What exactly causes the "login" to disappear from just adding these strings?
That's the answer, not the question. The very point of distinguishing bots from people is that the former should never (be able to) log in, only the latter. Otherwise bots would end up encountering the login again and again although it's obvious they have no credentials to enter.
Forex Station wrote:
Sat Aug 01, 2020 10:07 am
want to know if any of our current bot "strings" are affecting our site
Again: that's the very point that those definitions have an effect. It's also the very point to make them distinctive, as precise as possible. If you messed with it (as in: not understood how to formulate matches) then remove your adds again.

To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 4:13 pm
by Forex Station
AmigoJack wrote:
Sat Aug 01, 2020 12:34 pm

To make it clear: the 3 examples were no definitions, as version numbers and platforms in those texts will change over time. The proper definition for Ecosia as the most common denominator should be:

Code: Select all

 (Ecosia 
(select all text to see leading and trailing space)
Thanks for taking the time to write that up mate, I now understand it a bit better.

If in doubt, would adding a / at the end of the agent string help prevent any issues with strings?

For example, if i've got Google Pagespeed bot, this bot's string is
Chrome-Lighthouse. Seeing as this string has a browser name in it, and we shouldn't be adding browsers, would adding the forward slash at the end stop issues etc.?

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 5:42 pm
by AmigoJack
The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is. If you think (Ecosia / will match anything then you understood it all wrong. If you think Chrome-Lighthouse as a text that needs to match messes with the internet browser that only matches Chrome/ then you still understood it all wrong. Copy the 3 examples to your favorite editor, then hit CTRL+F and try to find matches what everything comes to your mind:
  • Chrome alone will match all 3 lines
  • Chrome-Lighthouse will match none
  • Mozilla/5.0 (Linux; Android 9; HMA-L29 Build/HUAWEIHMA-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.128 Mobile Safari/537.36 (Ecosia android@69.0.3497.128) will match only 1, but...
  • (Ecosia will match all 3 lines.
I thought it was more than obvious how searching text parts in texts works... We neither have regular expressions here, nor wildcards, nor any other magic or voodoo - only plain text. For what to find.

Re: Spiders & bots to add to phpBB

Posted: Sat Aug 01, 2020 5:58 pm
by Forex Station
AmigoJack wrote:
Sat Aug 01, 2020 5:42 pm
The / has no magic, it's a simple text match and the slash is a character just like E and the opening bracket is.
Understood. Thank you for clearing that up, got it now 👌