During the last week of February in 2018, several big internet sites started seeing a huge increase in a particular style of DDoS attack, taking advantage of the memcached protocol. Being the packet geeks we are, we wanted to explore the attack on one of our own internal servers and get a capture of what’s happening at the packet level so you can see it in action.
What is memcached?
The memcached protocol is a mechanism to assist load balancing the memory demands of large web services over multiple servers, so that a server not being used isn’t wasting memory. memcached is effective, and used by giants like Facebook to help manage the huge amounts of data and requests they have to deal with.
The basic operation of memcached is to receive a request for a particular “key” (i.e., database entry), and return the value to the user making the request.
Enter “memcrashed”
The problem is that memcached was never meant to be exposed to the Internet. The default memcached configuration allows connections from the network over TCP/UDP, but it is rarely used over UDP these days. That means it can often be overlooked as something that should be disabled - it’s not something most people would disable (and perhaps OS distributions should disable by default). Coupled with that, many default configurations listen for memcached messages on all interfaces.
Further, since there’s no authentication, anyone can connect to it and use it.
From memcached’s documentation:
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.
The attack works by exploiting an open system that is exposed to the Internet. If an attacker knows the IP address of its target, it can spoof the IP so that any memcached response goes to the target system rather than the attacker’s system.
An amplification attack vector
“memcrashed” is an amplification style DDoS attack, which means it escalates the amount of data going to a target. The way memcached works makes it particularly effective at building amplification attacks.
Cloudflare and Akamai have good write-ups on how this can be abused. From Akamai:
When a system receives a memcached get request, it forms a response by collecting the requested values from memory, sending them over the wire in an uninterrupted stream. This response is sent to the target in multiple UDP packets, each with a length of up to 1400 bytes. It is difficult to determine the exact amplification factor of memcached, but the attacks Akamai saw generated nearly 1 Gbps per reflector. Other organizations have reported attacks in excess of 500 Gbps using memcached reflection.
Attackers can also insert large data into the open server, giving a bigger chunk to amplify.
What does it look like?
Tom installed memcached on a test machine, using CentOS, with the default settings and left his firewall rules off exposing that memcached port to the rest of the network to show what can happen. Then he replicated the attack and grabbed captures of the whole thing. The basic process is:
1. Set a key with a large value on the vulnerable memcached server.
Tom sets 5 keys here of increasing bytes in size: 90, 812, 8,012, 80,013, and 800,013. We use this to illustrate how quickly this attack gets out of control. You can see a capture setting the keys here:
https://www.cloudshark.org/captures/df898a267665?filter=memcache
2. Spoof your IP address to that of the victim, and send memcached requests over and over, blasting the victim with UDP data.
Here’s a capture of the resulting attack. Tom requests each of the key values he set in step 1 five times:
https://www.cloudshark.org/captures/fe845cf4d01b?filter=ip.src%20%3D%3D%20172.16.0.62
If you look at a graph of the client and server showing the number of bytes sent by each, you can see that even the keys we used with small values sent an explosive amount of data to the target:
Quickly getting out of hand
Here’s a graph of the full attack. By the time we get to the largest values, the difference is so drastic you cannot even see the size of the client requests:
View the graph in CloudShark
Lastly, if you use the CloudShark Protocol Conversations tool, you can see the raw numbers. In this case we see 4.4 MB of UDP DDoS traffic created with only 1.6 KB of requests. That’s what they mean by amplification!
https://www.cloudshark.org/analysis/fe845cf4d01b/conversations?ladder=false
What can I do about it?
If you’re deploying memcached in your own infrastructure, it’s very important to pay attention to the default settings on any deployment so that you do not become a participant in one of these DDoS attacks. Especially make sure that anything you’ve set up in development gets locked down before you put it into production.
Here’s some takeaways:
- According to Cloudflare, shodan.io reports over 88,000 misconfigured servers.
- While it may not directly affect you, don’t contribute to the bad guys attacking other people!
- If you’re running memcached on your infrastructure, you should carefully check your configuration.
- The best way to lock it down is to disable UDP and only listen on localhost. Our friends at Digital Ocean have a great guide covering this.
- You should also check your own servers for open ports using something like nmap and tune your firewall accordingly.
- If your memcached instance is exposed, it is trivial for someone to flush all the records, which invalidates the entire point of memcached! It’s probably a good idea to make sure this isn’t exposed regardless of this particular exploit.
For some real world evidence of this attack, it looks like the software repository site GitHub got hit by this, and has an informative incident report on it.
Is my CloudShark server affected?
CloudShark uses memcached to speed up certain pages. If you’re a CloudShark Enterprise customer, you’ve followed our best-practices guide which covers setting up memcached to run only on localhost, as well as how to configure your firewall. CloudShark.org is locked down tight.
If you have any questions or concerns, contact support@cloudshark.io, we’d be happy to help.
Want articles like this delivered right to your inbox?
No spam, just good networking
