5 min read
During the last week of February in 2018, several big internet sites started seeing a huge increase in a particular style of DDoS attack, taking advantage of the memcached protocol. Being the packet geeks we are, we wanted to explore the attack on one of our own internal servers and get a capture of what’s happening at the packet level so you can see it in action.
The memcached protocol is a mechanism to assist load balancing the memory demands of large web services over multiple servers, so that a server not being used isn’t wasting memory. memcached is effective, and used by giants like Facebook to help manage the huge amounts of data and requests they have to deal with.
The basic operation of memcached is to receive a request for a particular “key” (i.e., database entry), and return the value to the user making the request.
The problem is that memcached was never meant to be exposed to the Internet. The default memcached configuration allows connections from the network over TCP/UDP, but it is rarely used over UDP these days. That means it can often be overlooked as something that should be disabled - it’s not something most people would disable (and perhaps OS distributions should disable by default). Coupled with that, many default configurations listen for memcached messages on all interfaces.
Further, since there’s no authentication, anyone can connect to it and use it.
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.
The attack works by exploiting an open system that is exposed to the Internet. If an attacker knows the IP address of its target, it can spoof the IP so that any memcached response goes to the target system rather than the attacker’s system.
“memcrashed” is an amplification style DDoS attack, which means it escalates the amount of data going to a target. The way memcached works makes it particularly effective at building amplification attacks.
Cloudflare and Akamai have good write-ups on how this can be abused. From Akamai:
When a system receives a memcached get request, it forms a response by collecting the requested values from memory, sending them over the wire in an uninterrupted stream. This response is sent to the target in multiple UDP packets, each with a length of up to 1400 bytes. It is difficult to determine the exact amplification factor of memcached, but the attacks Akamai saw generated nearly 1 Gbps per reflector. Other organizations have reported attacks in excess of 500 Gbps using memcached reflection.
Attackers can also insert large data into the open server, giving a bigger chunk to amplify.
Tom installed memcached on a test machine, using CentOS, with the default settings and left his firewall rules off exposing that memcached port to the rest of the network to show what can happen. Then he replicated the attack and grabbed captures of the whole thing. The basic process is:
Tom sets 5 keys here of increasing bytes in size: 90, 812, 8,012, 80,013, and 800,013. We use this to illustrate how quickly this attack gets out of control. You can see a capture setting the keys here:
Here’s a capture of the resulting attack. Tom requests each of the key values he set in step 1 five times:
If you look at a graph of the client and server showing the number of bytes sent by each, you can see that even the keys we used with small values sent an explosive amount of data to the target:
Here’s a graph of the full attack. By the time we get to the largest values, the difference is so drastic you cannot even see the size of the client requests:
View the graph in CloudShark
Lastly, if you use the CloudShark Protocol Conversations tool, you can see the raw numbers. In this case we see 4.4 MB of UDP DDoS traffic created with only 1.6 KB of requests. That’s what they mean by amplification!
If you’re deploying memcached in your own infrastructure, it’s very important to pay attention to the default settings on any deployment so that you do not become a participant in one of these DDoS attacks. Especially make sure that anything you’ve set up in development gets locked down before you put it into production.
Here’s some takeaways:
For some real world evidence of this attack, it looks like the software repository site GitHub got hit by this, and has an informative incident report on it.
CloudShark uses memcached to speed up certain pages. If you’re a CloudShark Enterprise customer, you’ve followed our best-practices guide which covers setting up memcached to run only on localhost, as well as how to configure your firewall. CloudShark.org is locked down tight.
If you have any questions or concerns, contact firstname.lastname@example.org, we’d be happy to help.
Want articles like this delivered right to your inbox?
No spam, just good networking