Claudebot attack

[ivailo95]

I have the same ClaudeBot attack and tried now with CF

Screenshot 2024-04-26 13.12.17.png

[Talk19Zehn]

Hello,

...(...)...
"helpful, honest, and harmless AI system" as they say it is.
We can skew their training by blocking access to the data.

Just a few notes from my point of view and experiences. The individual case decides, as you know.
So I guess we are swimming against the tide when they test, catch, check the AI ... They come in the middle of the night, some are caught (too many requests) and others may not and so on.
Yesterday there were 89 pages, this morning 66 pages (and unfortunately mostly from China). So I will probably have to "lock out" this country (?) ...

.htaccess: my own examples, tests

Code: Select all

RewriteCond %{HTTP_USER_AGENT} 11A465|Ahrefs|ArchiveBot|Baiduspider|BLEXBot|Bytedance|Bytespider|CCBot|ChatGPT-User|ClaudeBot|Curebot|Daum|Detectify|DotBot|Elisabot|Grapeshot|GPTBot|heritrix|Kinza|LieBaoFast|Linguee|LMY47V|MauiBot|Mb2345Browser|MegaIndex|MetaJobBot|MicroMessenger|MJ12bot|MQQBrowser|PageFreezer|PiplBot|Riddler|Screaming.Frog|Search365bot|SearchBlox|Seekport|SemanticScholarBot|SEOkicks|serpstatbot|Siteimprove.com|Sogou.web.spider|trendictionbot|TurnitinBot|UCBrowser|UptimeRobot|weborama-fetcher|Vagabondo|VelenPublicWebCrawler|YandexBot|YisouSpider [NC]
RewriteRule ^.* - [F,L]

I have since expanded the suggestion from P_I
# 28-Mar-24, P_I, another one to banish ->

I have since expanded it: my own examples, tests

Code: Select all

BrowserMatchNoCase "BLEXBot/1.0" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "ClaudeBot" bad_bot
BrowserMatchNoCase "ClaudeBot/1.0" bad_bot
BrowserMatchNoCase "Elisabot" bad_bot
BrowserMatchNoCase "UptimeRobot" bad_bot
BrowserMatchNoCase "UptimeRobot/1.0" bad_bot
BrowserMatchNoCase "UptimeRobot/2.0" bad_bot
Order Deny,Allow
Deny from env=bad_bot

And also added the robots.txt: my own examples, tests

Code: Select all

User-agent: BLEXBot/1.0
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: ClaudeBot/1.0
User-agent: Elisabot
User-agent: GPTBot
User-agent: UptimeRobot
User-agent: UptimeRobot/1.0
User-agent: UptimeRobot/2.0
Disallow: /

====
Well, I'm getting tired and thinking about giving it all up. Almost 20 years of phpBB are perhaps enough ....
Life is precious - best wishes

[xinu-mike]

How can you tell it is something like this?
For the past two weeks, my board has slowed to a crawl.
I have to purge all the sessions to get it to be snappy again.
Is this what is going on?
I am using version 3.3.10.

[HB]

If you define a new bot "ClaudBot" in the ACP > Spiders / Robots, that should tell you if it's the cause. I hadn't noticed a slowdown, but prompted by this topic, I checked and yes, that bot was making A LOT of requests. I added it to my blocked list on Cloudflare WAF (see How to prevent DDoS attacks? for details). Based on what I've seen in the events log, these so-called "AI Crawlers" are really aggressive about page requests.

If you're on a shared server and you're not behind a CDN, I can see how it would be easy for such bot activity to overwhelm your server.

[Prosk8er]

How can you tell it is something like this?
For the past two weeks, my board has slowed to a crawl.
I have to purge all the sessions to get it to be snappy again.
Is this what is going on?
I am using version 3.3.10.

you can tell by going to view who's online and view guests
in the user agent string you'd see claudebot in there

[xinu-mike]

Thank you.
I updated my .htaccess as suggested here and it has made a world of difference to the number of 'users' online.
I think I need to learn a bit more about what is going on, but this has been a help.

[xinu-mike]

I also see a lot of these: facebook external hit/1.1 as guests.
Are these a problem as well? I have like 100 of these guests.
Thank you.

[invenio]

I am also getting hit by this as well. Tried adding the bot blocking script into .htaccess. Let's see how it goes.

[Mick]

There’s an interesting topic here viewtopic.php?t=2550516

Also, if you Google ‘facebook external hit/1.1’ there’s a ton of information available and what steps to take etc. Personally if it were my board they’d be gone.

[P_I]

The good news is Claudebot seems to obey directives in robots.txt. Google turns up many websites having issues with Claudebot and taking action via robots.txt.

Therefore in my robots.txt I have added:

Code: Select all

# 20-Apr-24, P_I, Claudebot causing problems for phpBB sites
User-Agent: Claudebot
Disallow: /
User-agent: anthropic-ai
Disallow: /

I have kept the .htaccess handling in place for Claudbot as a backup.

[Ivanime]

Thank you.

I tried with .htacess and robots.txt, just in case, and the bot already stopped.

EDIT: since the changes, claudeBot returns at least 7 times in a day, but always checks robots.txt and leave the site

[ivailo95]

Every way to block bad bots work <3
Thanks, i have no problem with cloudebot, thank you !

by the way i remember a topic where can add bots is user group "bugs" for example.

[Mick]

Code: Select all

User-agent: UptimeRobot
User-agent: UptimeRobot/1.0
User-agent: UptimeRobot/2.0
Disallow: /

Instead of having three entries couldn’t you use a wild card like:

Code: Select all

User-agent: UptimeRobot/*

I don’t know, I’m just asking but will try it.

Edit: This morning I’m seeing what are presumably different versions of Claudebot:

User-Agent: ClaudeBot/1.0 etc.

Whereas I’ve seen no mention of Claudebot for days since robots.txt was installed.

[Talk19Zehn]

Hello Mick, my personal impression: at the moment, bots are indeed running amok.

AFAIK: UptimeRobot is an uptime monitoring service from the USA. They offer monitoring for websites, cronjobs and SSL.

I didn't understand why it suddenly appeared in my access logs. I don't use this type of monitoring separately, nor have I booked anything like this in the USA. I have also entered the bot in the .htaccess (see my quoted post above).

I have "banned" this bot via a test entry. Whether it helps is questionable. So far it hasn't come round again ....

Yes, try it out and report back.

Thanks and best regards

[Mick]

It seems the wildcard has worked as the newer versions of Claudebot havn’t been seen for an hour or two. Meanwhile, PetalBot is still hanging about, will see what happens.