Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
Post Reply
User avatar
Gwyneth Llewelyn
Registered User
Posts: 17
Joined: Thu Aug 06, 2009 11:34 pm
Location: Neufreistadt, Confederation of the Democratic Simulators, Second Life
Name: Gwyneth Llewelyn
Contact:

Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by Gwyneth Llewelyn » Sun Dec 02, 2018 7:55 pm

Hi folks,

This is neither a support request, neither a fix, but more a report on what happened to one of the forums that I maintain earlier today: a Distributed-Denial-of-Service attack, originating in China, with no purpose I could figure out except to put the forums down.

So, how was this accomplished? In fact, they used a very simple strategy: they attempted to log in from a group of servers (I counted manually around two dozens, but there were possibly more), all coming from the 220.243.0.0/16 range, 'faking' a different User-Agent tag each time (so that it always looked like a different attempt, even if coming from the same IP address). They timed the requests carefully so that they didn't hit any request limits — from the perspective of the web server, it 'seemed' that different requests would come from different machines (or different browsers under the same address) at a 'reasonable' rate, say, just one per second, which would go through most kinds of protective measures.

Now, my nginx/PHP setup can most definitely handle one request per second or even two dozen requests per second — that would really be way within the threshold. The problem was that each of those requests would require opening a new session (with its own cookie, etc.). Each session will 'hit' the database at least once, to figure out if it's the 'same' request or not (in this case, it wouldn't be the same) or if a new session had to be created. Very quickly this grew to a million open sessions or more; however, this was not the real issue, MySQL/MariaDB can most certainly handle tables with millions of entries.

No, the problem was that each time a session was created, there is a check made to see how many sessions already exist for this particular user (SELECT COUNT(session_id) [...] FROM [table-prefix]sessions) and update these accordingly. With a 'usual' amount of open sessions — say, a dozen, a hundred, even some twenty thousand, as I've seen in some really large forums — this is handled by the database in nanoseconds (well... perhaps microseconds). But if you start having a million open sessions, and have to count them each time, and before the count is finished, a new request pops up requiring the count operation to be repeated... well, you get the idea. At some point, no matter how much optimisation is in your system, no matter how many results are being cached (because each new session will get a different result, there will be a limit to how many operations can actually be cached — I seriously suspect that 'close to none', except in those very few cases when 2 or 3 requests come in before any of them has created a new session, in which case the count will be the same), the database is basically doing nothing else but counting sessions over and over and over, and as the number of requests for new sessions are coming in more and more, it basically 'stalls' because it cannot proceed further — and that means that the code cleaning up the unused sessions will never get a chance to run: the sessions table will grow faster than the cleaning-up code can do its job; and, in any case, the database will be way too busy counting the number of sessions instead of worrying about anything else.

You can take a look at what's going on inside the file phpbb/sessions.php, around lines 778 and further around 838 (for phpBB 3.2.4). You might notice something already very cleverly done by the core programmers: the very same problem might have been triggered by 'legitimate' bots, namely those which are indexing content for Google, Bing, and similar services. Those bots use well-known User-Agents, so the code can be optimised for them: in their case, for example, phpBB considers them to be a 'special' anonymous user, and reuses sessions (instead of creating a new one every time those bots connect). This works quite well, and I presume that as more legitimate bots pop up, the core programmers will happily add them to the database table.

To prevent the DDoS attack I got, however, something else needs to be done code-wise. Remember that the trick that these cyberbullies did was to rotate requests among a few dozen servers, making sure that each IP would make only 'reasonable' amounts of requests, but using different User-Agent handles for each request. From the perspective of phpBB, it has no idea that these are anything but 'legitimate' users of some sort, so all these requests were honoured. To make matters even worse, from the perspective of the nginx/PHP setup, the amount of requests was also considered to be 'reasonable' for all purposes, since they seemed to be genuine requests coming from different IP addresses and different User-Agents (e.g. different browsers on different devices). Thus, filtering at that level, the attacks would simply go through; even limiting the rate of requests to a very low threshold did little to help (especially after the sessions table already had over a million entries), because, as a matter of fact, it would be far more likely to restrict legitimate users (take into account that each legitimate browser will require several requests, for downloading images, CSS, and JS...) than the illegitimate attacks.

In my setup, I have CloudFlare in front of everything, to deal exactly with these kinds of things. But, of course, even CloudFlare is not sufficiently smart to 'detect' that this was an 'attack' and not merely a lot of genuine traffic. It required a human (yours truly) to block the 220.243.0.0/16 range and escalate the level of defence to 'Under attack' mode, which triggers an irritating additional page where the browser is checked to see if there is a human behind it, and if so, allows it through the website. This effectively blocked the attack. I left the settings like that for several hours, because, as you may imagine, besides this very specific DDoS attack, there are always — every day, every hour — some typical spambots trying to get access to our forums (with little success). These, however, are not that 'harmful' — after a few tries without success, the spammer simply moves to the next forum on their list, and gives up on those which are too hard to break into...

Now, how could this DDoS attack be prevented?

I'm not a professional programmer by any account, so I'm unfamiliar with the best approach to this issue, but I can imagine that something could be done to limit the impact of 'fake' requests on the database. And this would be to at least locally cache the count of open sessions: after all, if we're talking about humans, it's unlikely — except on the busiest forums! — that hundreds of new sessions would be created at the same time, so it would make sense to cache the result of the SELECT COUNT(session_id) [...] command, for, say, one second. One second makes a huge difference — it's almost negligible for a human, but it means quite a lot for modern CPUs and databases which work at the microsecond level.

It would also make sense to limit the number of open sessions to a reasonable default — this could be a configuration parameter. Remember that there are three different cases here: the first is a legitimate user, which opens a new session but really uses it. Then there are the legitimate content-indexing bots, which get the 'same' session using the already-existing trick implemented in the code. And finally, there is the case of someone opening a session but failing to put their password and actually logging in, which could be much more restricted than it is. We have, after all, limits in place for the number of failed logins. However, in the scenario I've described, the problem was not with people logging in, but rather with 'anonymous usage' of the forums — people who just want to see the public areas of the forums without having any intention of becoming regular users — and these could theoretically be limited to a reasonable default. Hundreds or perhaps even thousands of simultaneous anonymous users would be fine in most setups; a million would certainly not be acceptable. So having a reasonable default — say, 100 — would make a lot of sense.

In the case of a future DDoS attack like the one I suffered, what would simply happen would be that the first 100 requests would be promptly answered, but the 101st would be refused and the connection shut down, no matter who tries to see the content (note that legitimate, logged-in users or content-indexing bots would not be affected, but only anonymous users). This might be acceptable in many communities. Others, however, wish to have as many viewers (registered or not) as possible, in order to leverage ads to get revenue; again, such forums might wish a much higher number of simultaneous users, each with a valid, open session. But I find it hard to believe that there is a scenario where a forum admin really wants a million, or ten million, of simultaneous anonymous users. Nevertheless, that option might be possible, with a fair warning that allowing too many anonymous users.

Note that there is already a fair amount of configuration that can be done for the 'Anonymous' user. However, AFAIK, there is no way to configure a specific amount of maximum anonymous users that are allowed to be simultaneously logged in, but I guess that this ought to be a relatively simple thing to implement... not for me, though, I'm simply not familiar enough with the phpBB code to do that!

Also note that I'm not really requesting anything, just opening the discussion to see what you guys think — namely, how serious these kinds of DDoS attacks really are, in the sense of being worth preventing them by adding some extra checks to the code. Maybe these are so rare — compared to the effort in preventing spam! — that it's worthless to change the core code to deal with this very specific case; maybe it justifies an extension or something else but not an actual change to the code. But I think it's still worth reflecting on the issue simply because anything in the code that can be abused in order to bring a system down is, IMHO, worth preventing!

User avatar
Gwyneth Llewelyn
Registered User
Posts: 17
Joined: Thu Aug 06, 2009 11:34 pm
Location: Neufreistadt, Confederation of the Democratic Simulators, Second Life
Name: Gwyneth Llewelyn
Contact:

Re: Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by Gwyneth Llewelyn » Sun Dec 02, 2018 8:04 pm

Just an additional bit of clarification: this kind of attack is known as an HTTP Flood DDoS (see https://www.cloudflare.com/learning/ddo ... os-attack/ and jump to the HTTP Flood section).

User avatar
canonknipser
Registered User
Posts: 1713
Joined: Thu Sep 08, 2011 4:16 am
Location: Germany
Name: Frank Jakobs
Contact:

Re: Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by canonknipser » Sun Dec 02, 2018 9:14 pm

I don't think it was a DDos attack - just a group aggressive search bot following google's "Mobile first" strategy. It happen a few days ago on other sites, see also this topic: viewtopic.php?f=556&t=2491366
Greetings, Frank
phpbb.de support team member
English is not my native language - no support via PM or mail
New arrival - Extensions and scripts for phpBB

User avatar
david63
Jr. Extension Validator
Posts: 15070
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by david63 » Sun Dec 02, 2018 9:47 pm

This is nothing new, this Chinese attack has been doing the rounds for at least the last month
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Gwyneth Llewelyn
Registered User
Posts: 17
Joined: Thu Aug 06, 2009 11:34 pm
Location: Neufreistadt, Confederation of the Democratic Simulators, Second Life
Name: Gwyneth Llewelyn
Contact:

Re: Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by Gwyneth Llewelyn » Sun Dec 02, 2018 10:21 pm

All right, thanks for the clarification. Aye, I can confirm that the range of incoming requests was from the 220.243.0.0/16 block (about 7852 unique addresses, as far as I can count them), so it's fair to admit that it's the same kind of thing as described on the thread you mentioned.

On the flip side of the coin... where exactly do you draw the line between an HTTP Flood attack and a 'group aggressive search bot' using whatever strategy — especially if the end result is a Denial of Service? My point here was not precisely to discuss if this is something 'new', if it's a deliberate attack (in the sense that, say, the Chinese government wants that particular forum — which just happens to be about democracy and governance! — to deliberately be brought down), or if it's just a side-effect from a 'legitimate' search engine which is way too aggressive..

As far as I could find on the Internet, that specific IP address block is not associated with any 'popular' search engine...

User avatar
John connor
Registered User
Posts: 1836
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Contact:

Re: Suffered a DDoS attack - simple in concept and (hopefully) easy to fix?

Post by John connor » Mon Dec 03, 2018 2:31 am

Two things:


Since you need an upgraded account in CloudFlare to do this, block the country of China in CloudFlare. If it's a free account, use the captacha.

In CloudFlare there is a layer 7 HTTP DDoS mitigation setting under Firewall. I use it myself and it costs me about 30 cents a month.

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: No registered users and 22 guests