Claudebot attack

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
ivailo95
Registered User
Posts: 1097
Joined: Tue Sep 05, 2017 8:00 am
Location: Bulgaria
Name: Ivailo

Re: Claudebot attack

Post by ivailo95 »

I have the same ClaudeBot attack and tried now with CF
Screenshot 2024-04-26 13.12.17.png
You do not have the required permissions to view the files attached to this post.
For REALLY good and VERY cheap hosting CLICK HERE
Watch "Lost in a random" game, it's the best
User avatar
Talk19Zehn
Registered User
Posts: 903
Joined: Tue Aug 09, 2011 1:10 pm

Re: Claudebot attack

Post by Talk19Zehn »

Hello,
lurttinen wrote: Thu Apr 25, 2024 12:19 pm ...(...)...
"helpful, honest, and harmless AI system" as they say it is.
We can skew their training by blocking access to the data.
Just a few notes from my point of view and experiences. The individual case decides, as you know.
So I guess we are swimming against the tide when they test, catch, check the AI ... They come in the middle of the night, some are caught (too many requests) and others may not and so on.
Yesterday there were 89 pages, this morning 66 pages (and unfortunately mostly from China). So I will probably have to "lock out" this country (?) ...

.htaccess: my own examples, tests

Code: Select all

RewriteCond %{HTTP_USER_AGENT} 11A465|Ahrefs|ArchiveBot|Baiduspider|BLEXBot|Bytedance|Bytespider|CCBot|ChatGPT-User|ClaudeBot|Curebot|Daum|Detectify|DotBot|Elisabot|Grapeshot|GPTBot|heritrix|Kinza|LieBaoFast|Linguee|LMY47V|MauiBot|Mb2345Browser|MegaIndex|MetaJobBot|MicroMessenger|MJ12bot|MQQBrowser|PageFreezer|PiplBot|Riddler|Screaming.Frog|Search365bot|SearchBlox|Seekport|SemanticScholarBot|SEOkicks|serpstatbot|Siteimprove.com|Sogou.web.spider|trendictionbot|TurnitinBot|UCBrowser|UptimeRobot|weborama-fetcher|Vagabondo|VelenPublicWebCrawler|YandexBot|YisouSpider [NC]
RewriteRule ^.* - [F,L]
I have since expanded the suggestion from P_I
# 28-Mar-24, P_I, another one to banish ->

I have since expanded it: my own examples, tests

Code: Select all

BrowserMatchNoCase "BLEXBot/1.0" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "ClaudeBot" bad_bot
BrowserMatchNoCase "ClaudeBot/1.0" bad_bot
BrowserMatchNoCase "Elisabot" bad_bot
BrowserMatchNoCase "UptimeRobot" bad_bot
BrowserMatchNoCase "UptimeRobot/1.0" bad_bot
BrowserMatchNoCase "UptimeRobot/2.0" bad_bot
Order Deny,Allow
Deny from env=bad_bot
And also added the robots.txt: my own examples, tests

Code: Select all

User-agent: BLEXBot/1.0
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: ClaudeBot/1.0
User-agent: Elisabot
User-agent: GPTBot
User-agent: UptimeRobot
User-agent: UptimeRobot/1.0
User-agent: UptimeRobot/2.0
Disallow: /
====
Well, I'm getting tired and thinking about giving it all up. Almost 20 years of phpBB are perhaps enough ....
Life is precious - best wishes
xinu-mike
Registered User
Posts: 9
Joined: Sat Jan 28, 2023 12:44 pm

Re: Claudebot attack

Post by xinu-mike »

How can you tell it is something like this?
For the past two weeks, my board has slowed to a crawl.
I have to purge all the sessions to get it to be snappy again.
Is this what is going on?
I am using version 3.3.10.
HB
Registered User
Posts: 230
Joined: Mon May 16, 2005 9:30 pm

Re: Claudebot attack

Post by HB »

If you define a new bot "ClaudBot" in the ACP > Spiders / Robots, that should tell you if it's the cause. I hadn't noticed a slowdown, but prompted by this topic, I checked and yes, that bot was making A LOT of requests. I added it to my blocked list on Cloudflare WAF (see How to prevent DDoS attacks? for details). Based on what I've seen in the events log, these so-called "AI Crawlers" are really aggressive about page requests.

If you're on a shared server and you're not behind a CDN, I can see how it would be easy for such bot activity to overwhelm your server. :|
Dan Kehn
User avatar
Prosk8er
Registered User
Posts: 1764
Joined: Sun Mar 12, 2006 3:30 am
Location: Rochester, NY
Name: Tyler

Re: Claudebot attack

Post by Prosk8er »

xinu-mike wrote: Fri Apr 26, 2024 9:02 pm How can you tell it is something like this?
For the past two weeks, my board has slowed to a crawl.
I have to purge all the sessions to get it to be snappy again.
Is this what is going on?
I am using version 3.3.10.
you can tell by going to view who's online and view guests
in the user agent string you'd see claudebot in there
xinu-mike
Registered User
Posts: 9
Joined: Sat Jan 28, 2023 12:44 pm

Re: Claudebot attack

Post by xinu-mike »

Thank you.
I updated my .htaccess as suggested here and it has made a world of difference to the number of 'users' online.
I think I need to learn a bit more about what is going on, but this has been a help.
xinu-mike
Registered User
Posts: 9
Joined: Sat Jan 28, 2023 12:44 pm

Re: Claudebot attack

Post by xinu-mike »

I also see a lot of these: facebook external hit/1.1 as guests.
Are these a problem as well? I have like 100 of these guests.
Thank you.
User avatar
invenio
Registered User
Posts: 414
Joined: Wed Dec 09, 2015 1:45 pm
Location: New Hampshire, USA

Re: Claudebot attack

Post by invenio »

I am also getting hit by this as well. Tried adding the bot blocking script into .htaccess. Let's see how it goes.
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26849
Joined: Fri Aug 29, 2008 9:49 am

Re: Claudebot attack

Post by Mick »

There’s an interesting topic here viewtopic.php?t=2550516

Also, if you Google ‘facebook external hit/1.1’ there’s a ton of information available and what steps to take etc. Personally if it were my board they’d be gone.
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧
User avatar
P_I
Community Team Member
Community Team Member
Posts: 2462
Joined: Tue Mar 01, 2011 8:35 pm
Location: Western Canada 🇨🇦

Re: Claudebot attack

Post by P_I »

The good news is Claudebot seems to obey directives in robots.txt. Google turns up many websites having issues with Claudebot and taking action via robots.txt.

Therefore in my robots.txt I have added:

Code: Select all

# 20-Apr-24, P_I, Claudebot causing problems for phpBB sites
User-Agent: Claudebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
I have kept the .htaccess handling in place for Claudbot as a backup.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
User avatar
Ivanime
Registered User
Posts: 45
Joined: Fri Oct 03, 2003 8:31 pm
Location: Santiago, Chile

Re: Claudebot attack

Post by Ivanime »

Thank you.

I tried with .htacess and robots.txt, just in case, and the bot already stopped.

EDIT: since the changes, claudeBot returns at least 7 times in a day, but always checks robots.txt and leave the site
Last edited by Ivanime on Fri May 03, 2024 12:36 am, edited 1 time in total.
User avatar
ivailo95
Registered User
Posts: 1097
Joined: Tue Sep 05, 2017 8:00 am
Location: Bulgaria
Name: Ivailo

Re: Claudebot attack

Post by ivailo95 »

Every way to block bad bots work <3
Thanks, i have no problem with cloudebot, thank you !

by the way i remember a topic where can add bots is user group "bugs" for example.
For REALLY good and VERY cheap hosting CLICK HERE
Watch "Lost in a random" game, it's the best
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26849
Joined: Fri Aug 29, 2008 9:49 am

Re: Claudebot attack

Post by Mick »

Talk19Zehn wrote: Fri Apr 26, 2024 1:33 pm

Code: Select all

User-agent: UptimeRobot
User-agent: UptimeRobot/1.0
User-agent: UptimeRobot/2.0
Disallow: /
Instead of having three entries couldn’t you use a wild card like:

Code: Select all

User-agent: UptimeRobot/*
❓

I don’t know, I’m just asking but will try it.

Edit: This morning I’m seeing what are presumably different versions of Claudebot:

User-Agent: ClaudeBot/1.0 etc.

Whereas I’ve seen no mention of Claudebot for days since robots.txt was installed.
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧
User avatar
Talk19Zehn
Registered User
Posts: 903
Joined: Tue Aug 09, 2011 1:10 pm

Re: Claudebot attack

Post by Talk19Zehn »

Hello Mick, my personal impression: at the moment, bots are indeed running amok.

AFAIK: UptimeRobot is an uptime monitoring service from the USA. They offer monitoring for websites, cronjobs and SSL.

I didn't understand why it suddenly appeared in my access logs. I don't use this type of monitoring separately, nor have I booked anything like this in the USA. I have also entered the bot in the .htaccess (see my quoted post above).

I have "banned" this bot via a test entry. Whether it helps is questionable. So far it hasn't come round again ....

Yes, try it out and report back.

Thanks and best regards
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26849
Joined: Fri Aug 29, 2008 9:49 am

Re: Claudebot attack

Post by Mick »

It seems the wildcard has worked as the newer versions of Claudebot havn’t been seen for an hour or two. Meanwhile, PetalBot is still hanging about, will see what happens.
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧

Return to “phpBB Discussion”