file.php using 100% CPU ?

exxos · Post by **exxos** » Sat Jun 03, 2023 10:13 am

I am running Ubuntu 20.04 with Apache 2.57. PHPBB 3.3.4. MySQL(i) 8.0.33. PHP 7.4.3-4ubuntu2.18

In my Apache server status page I can see the connections which appear to be "stuck" using near 100% CPU. Though the bank of children also seems to get stuck near 100% CPU.

The main server CPU is currently reporting a maxed out core. This progressively gets worse until the second core gets maxed out as well. Then the forum starts running incredibly slow.

There seems to be no pattern to it as such.

If I ignore all are currently closed connections, all the high CPU usage ones always seem to be with the download/file.php script.

I've also opened a bug report with Apache but I don't think I will get anywhere in that respect. But as the pages seem to be always on the same page with PHPBB, I thought I would ask if anyone else the same issue or is this already a known issue which was solved in later PHPBB versions ?

Incidentally the files themselves are very small images mostly. Some of them are around 2MB because of animated GIFs. But I checked several others and they were simply just static small images around 640x480 size jpgs / png etc. So nothing demanding there at all.

janus_zonstraal · Post by **janus_zonstraal** » Sat Jun 03, 2023 4:07 pm

Probably a lot of bots and scrapers:

1 hidden and 786 guests

Post by **Mick** » Sat Jun 03, 2023 4:17 pm

exxos wrote: ↑Sat Jun 03, 2023 10:13 amis this already a known issue which was solved in later PHPBB versions ?

What makes you think it’s a phpBB problem to be solved?

HB · Post by HB » Sat Jun 03, 2023 4:56 pm

That IP address does look like a bot/scraper https://whatismyipaddress.com/ip/18.142.215.173 (southeast-1.compute.amazonaws.com). The user agent might help prove/disprove that theory, though user agents are easily spoofed. That said, I cannot think of a reason file.php would peg the CPU for a single invocation.

janus_zonstraal · Post by **janus_zonstraal** » Sat Jun 03, 2023 5:59 pm

786 guest online, is that realistic?

exxos · Post by **exxos** » Sat Jun 03, 2023 6:10 pm

There is a lot of bots but they only doing a request every few seconds. The server is doing about 7 requests a second. It's nothing.

However, one connection has been open on file.php for the past few hours doing nothing but using high CPU.

I don't know if it's a phpbb problem or not. This is why I'm asking for help and advice. Nothing else on my website is stuck, only the phpbb requests. Until someone suggests otherwise, I have to start somewhere don't I ?

There are plenty of open server connections , plenty of ram, network traffic is almost zero.

janus_zonstraal · Post by **janus_zonstraal** » Sat Jun 03, 2023 7:03 pm

I have to start somewhere don't I ?

So why not starting with the most officious solution and block bad bots that eating your resources?

HB · Post by HB » Sat Jun 03, 2023 8:05 pm

exxos wrote: ↑Sat Jun 03, 2023 6:10 pmHowever, one connection has been open on file.php for the past few hours doing nothing but using high CPU.

You lost me there.

Are there multiple requests (as indicated in your Apache access log) or a single connection that remains open for a long time (if so, doing what?)? If the latter, I would start by configuring PHP max_execution_time and double-checking your Apache configuration (e.g.,KeepAlive Off).

It would also be helpful to have some logs from file.php, for example:

Code: Select all

$log_text = "Blah blah message + key parameters";
$msg_title = "Debug information for file.php at line " . __LINE__;
$phpbb_log->add('critical', $user->data['user_id'], $user->ip, 'LOG_GENERAL_ERROR', false, array($msg_title, $log_text));

Sprinkle a few of these in file.php and you'll at least have an idea of what was being requested, by whom, and how frequently.

exxos · Post by **exxos** » Sun Jun 04, 2023 12:30 am

Thanks for your suggestions. PHP timeouts and Apache timeouts have all been set. Connections are dropped in under 60 seconds. But some requests to file.php never seem to close in some cases. The CPU time in apache server-status just keeps going up but it seems "stuck" on the same file.

The IPs are coming from 54.255.254.92 , 18.138.116.110 , Once I forced them closed, the CPU dropped to almost nothing again. One was stuck on "W" the other "G".

I think this in part to a DoS attack I had here viewtopic.php?p=15940466#p15940466

But it was only those 2 IPs. Apache deals with about 5 connections, confirmed in all the logs that the bots are only doing 1 connection a second. They are a few bots, but Apache reports about 2-8 connections a second. The server is basically doing nothing.

I don't know what amazonaws.com is doing, it doesn't seem to be requesting multiple files or anything, just one from file.php, connection never closes and it maxes out the server CPU. I don't get it.

I will turn off "keepalive" in Apache tomorrow and see if that helps.

Though is there any way to block amazonaws.com from accessing the forum at all within phpbb ? I'm, not sure if "bad bots" will cover this problem ?

HB · Post by HB » Sun Jun 04, 2023 1:07 am

Holding a connection for 60 seconds seems like a very long time. After all, we're talking about a browser, so holding a connection for more than a few seconds seems patient enough to me. I don't see how "some requests to file.php never seem to close" is possible; after all, the PHP execution time will kill the process and that closes the TCP/database connection.

As for these:

54.255.254.92: https://whatismyipaddress.com/ip/54.255.254.92 (Amazon AWS Japan)
18.138.116.110: https://whatismyipaddress.com/ip/18.138.116.110 (Amazon AWS Singapore)

These aren't users, that's for sure. My assumption is they're bots or scrapers. I would not feel guilty about blocking them.

If it's a DDoS (and not a bug), one sure way to end it is with Cloudflare and a Web Application Firewall (see this post for details). You can block by IP address, IP address range, country, user agent, etc. Cloudflare acts as the front end to your server ("host" in their terms) and can filter traffic. If you want to try the blunt axe approach, you can flick the switch "I'm under attack!" and Cloudflare will challenge every access with a CAPTCHA-like challenge. There are less dramatic settings that only block known malicious bots, challenge depending on the originating country, etc. If you want to get fancy, you can even control these modes programmatically (e.g., if CPU spikes above X for > N minutes, switch to "I'm under attack" mode for M minutes).

Note that you can use phpBB's native "bad bots" blocking mechanisms, but that still creates an incoming session, accesses the database several times, etc. It's not hard for a malicious botnet to tip over your server simply by creating sessions in rapid succession. Blocking with .htaccess can prevent them from creating sessions. But by putting your (host) server behind a proxy, the attackers have to get past its challenges before your server even sees a TCP connection request.

exxos · Post by **exxos** » Sun Jun 04, 2023 8:18 am

HB wrote: ↑Sun Jun 04, 2023 1:07 am Holding a connection for 60 seconds seems like a very long time. After all, we're talking about a browser, so holding a connection for more than a few seconds seems patient enough to me. I don't see how "some requests to file.php never seem to close" is possible; after all, the PHP execution time will kill the process and that closes the TCP/database connection.

Yes absolutely, that is why I don't get how a connection is constantly using CPU for hours while seemingly doing nothing.

Last night I killed the two connections and the CPU dropped to zero pretty much. A few minutes later and the CPU ramped up again. Has a quick fix I just simply renamed file.php and the server remains functioning fine with near zero CPU usage.

One connection had been using CPU power for over 100,000 seconds and Apache only shows the same file.php . It was also stuck on gracefully finishing but never actually did.

Cloudflare I'm reluctant to use because I have had constant problems with such services blocking legitimate traffic. Im I am not sure if they're under attack mode will work because I only get about one connection a second on average. That was just literally two connections which were not serving any content but consuming 100% CPU on a file.php link.

Incidentally I have been checking that those links are actually legitimate and they do indeed load images which have been posted on the forum. The connection serves and closes as expected.

Post by **Mick** » Sun Jun 04, 2023 8:22 am

Did you try contacting Amazon AWS abuse as per viewtopic.php?p=15940481#p15940481?

exxos · Post by **exxos** » Sun Jun 04, 2023 9:18 am

Mick wrote: ↑Sun Jun 04, 2023 8:22 am Did you try contacting Amazon AWS abuse as per viewtopic.php?p=15940481#p15940481?

I haven't no. I kind of gave up bothering contacting companies about issues. I only tend to speak up these days if I really have no other choice. I have been at this current problem for the past few weeks now

Cloudflare and microsoft I have had long terms issues with and gave up after microsoft. They were blocking my server IP, had no idea why (they even told me that) but they were very adamant they were going to continue to do so. Cloudflare kept blocking my IP, I unblocked it, next day, same again. After several loops of that, I gave up. I also heard Cloudflare and microsft were causing issues for other people on my forum with their own services. There is simply not enough hours in the day to deal with all these problems/companies constantly. It always ends the same way, wishing I simply not had bothered.

Anyway, it was only 2 IPs causing the problems last night from AWS. I kind of think its some Apache problem which is triggered when AWS is accessing file.php. There are 900 bots on my forum and currently everything is fine and has been for the past several hours. So I just enabled file.php again. I suspect later today the high CPU problems will return. Its not a "high traffic" problem. Though I have read there can be "Slow DoS attacks" a few days ago.

Problem is , I don't know how to recreate this problem as such. So I suspect Apache will close the issues as "can't replicate" or blame PHPBB and close the ticket. Either way, I have reported the issue in case someone else has come across the same problem in the future.

Its why I asked if there was any known compatibility issues with the versions of the software I am using as I am a few versions behind with PHPBB. When I looked a few days ago, the bug fixes only said something like "fixed various PHP8 issues" but didn't say what they were. I haven't had chance to look if any changed in file.php to see if that had changed recently for any reason. Though I assume not anyway.

exxos · Post by **exxos** » Sun Jun 04, 2023 10:17 am

I just noticed this.

Code: Select all

0-1	178852	8/1650/3448	W	77.72	82	42	139797	272.1	15.04	55.22	13.215.58.28	h2	exxosforum.co.uk:443	[8/5] write: stream 13, GET /forum/download/file.php?id=25389&s

Does anyone know if writes to file.php are normal ?

That IP comes back as (compatible; Bytespider; https://zhanzhang.toutiao.com/)" in the logs.

Interestingly:

Code: Select all

13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23635&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23656&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23653&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23654&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23657&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23659&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23655&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23661&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23660&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/images/ranks/admin2.png HTTP/2.0" 200 2864 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23658&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23662&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"
13.215.58.28 - - [04/Jun/2023:03:56:15 +0100] "GET /forum/download/file.php?id=23530&sid=168804b5de10340a2c463c73fcbb3b34 HTTP/2.0" 404 530 "https://www.exxosforum.co.uk/forum/viewtopic.php?p=78772&sid=019da78e2cd8f4955f59fbdaa8cb58c3" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)"

This is opending a lot of connections per second it seems

These wasn't in the logs a few days ago, so seems to be something new.

janus_zonstraal · Post by **janus_zonstraal** » Sun Jun 04, 2023 10:51 am

Did you block the bots already?