Some bots not detected ?

[exxos]

I have two forums and I set up the botd list yesterday on the version with 3.3.4.. And it correctly detects the bots I added..

The one in question is amazon bot.. On a recently updated second forum using the latest forum pack.. I deleted out amazon bot on it and added it back in and is enabled etc but it never shows up in the online list ?

However, currently Google and Bing have shown up in the online list... So I am somewhat baffled why amazon bot isn't... Its basically hitting the server once a second and has been for hours...

It is puzzling because I am getting the "sid=" which was why I started adding in bots as some were getting stuck in a loop as sid kept changing.. Once I added in the missing bots into my 3.3.4 install, all those problems went away. But on the latest forum install, amazon bot seems to simply not found at all.

Anyone have any ideas what is it going on or is this a bug in the later forum installs ?

1.PNG

2.PNG

3.PNG

[exxos]

The strange thing is, if I do

Code: Select all

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)" -I https://exxosforum.co.uk/AMN/search.php?search_id=active_topics&sid=7069e1285bb2076217a57499731a746f

Amazon bot then shows up..

2.PNG

[exxos]

It seems my board time zone was wrong... I set it to the correct one and now Amazon bot seems be behaving now but the online time in the online list is still wrong..

11.PNG

22.PNG

I did clear the sessions and cache but did not help. Time is correct on my other forum.

So my user profile time is the likely cause, seems ok now i've changed that. So problem solved I guess..

[exxos]

Annoyingly the problem has returned with Amazon bot. Loads of entries in my logs, but not showing up on the bots list on the forum again. Other bots are listed still.

I don't have this problem on my slightly older forum 3.3.4. So I don't know if something in bot detection has changed in later versions of phpbb ? Or there is something else going on, but no idea what

[Brf]

Is it being matched to another bot for some reason?
I see in one of your images that you have Google bot and apple bot on the same session.

[exxos]

Is it being matched to another bot for some reason?

I don't believe so...

I looked in the SQL DB directly for amazonbot and its only listed once. I searched for "zon" as well and only one result returned.

I see in one of your images that you have Google bot and apple bot on the same session.

weird..

ahhhhh

1.PNG

2.PNG

Applebot !

So assume I am dumb as usual at this point... will change it and see if it behaves again this time.. Thanks!

[exxos]

Still have the same problem somehow For some bizarre reason amazonbot is no longer on my main forum which had been working.

All the other bots are visiting both forums as normal.

Amazon bot is still being logged about once a second in the nginx access.log. So it's like the spiders/bots rule never matched amazonbot..

Code: Select all

34.236.135.14 - - [01/Oct/2024:16:51:21 +0100] "GET /AMN/index.php?sid=ff0e528f3ef1bc4ea05192a819cab20a HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"

1.PNG

Other bots are very active also, so I can't see it being anything server related, at least what I can think of.

semrush bot seems fine for example..

Code: Select all

85.208.96.212 - - [01/Oct/2024:17:58:21 +0100] "GET /forum/viewtopic.php?t=6193 HTTP/1.1" 200 9841 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"

11.PNG

[exxos]

I think I might know what one problem is...

Amazonbot doesn't seem to respect the robots.txt rate limit very well... So is actually triggering my rate protection rules of 20 connections per second per IP.. Amazonbot has multiple IPs, so that amount of traffic can quickly become a problem for small servers.

I removed that jail, amazon bot has appeared on my main forum (once ) , but still not appearing on my second smaller forum.. But still in the logs..

It's possible amazon bot is also hitting DDOS type protection as well. But nothing is banned..

So I assume this is a server related problem... So even though it's not a solved problem.. maybe my waffle may help someone else in the future anyway..

[exxos]

Another thought ..Is it possible Amazonbot has all links with the (now) obsolete SID's, and in that case, would phpbb cause a 301 redirect on invalid SID's ? and thus not be found in phpbb bots detection anymore ?

[Mick]

Amazonbot doesn't seem to respect the robots.txt

In that case I suggest you try the .htaccess method as described in the Claudebot attack topic.