I wake up yesterday morning and while still in bed I get the dreaded site-down alert from Pingdom on my smartphone. When a Web site goes down there are a number of simple preliminary steps one takes to pinpoint and fix the problem. Is the ISP having an outage? Are the modem and router up? Is the server up and is the Web service running?
The server was up but the Web service was unresponsive. The quick and dirty steps are restart the Web service, no dice there. Ok, reboot the server, still no good. It was time do drag myself over to my desk and login to the Linux server to investigate more. Going through a bunch of diagnostics steps, this is what I saw:
The Flood of The SYN-tury
Those familiar with the TCP handshake know that the session setup consists of a SYN packet from host to server, followed by a SYN-ACK packet from server to host and finally a ACK from the host to server and the connection is established. When one sees reams of SYN_RECV on the server it is indicative of a possible attack where a host or a group of them flood the server with the first SYN but they spoof their IP addresses or just snub the server's SYN-ACK packet saddling the server with these half-open connections, each one taking up a bit of the server's resources.
The server eventually cleans up the half-open connections but if the zombie connection numbers rise too fast, they exhaust the server and no more connections are accepted; the server goes offline. This is known as a SYN flood attack and it eventually leads to a condition known as DoS (Denial of Service) or the more dreaded form, DDoS (Distributed Denial of Service).
By now I was late for work, so I just blocked all traffic to the server and went to my day job. The server remained crippled all day (apologies to the users of my utilities) until I returned home and began the process of resolving the issue. I was hoping that by then the attackers had moved on to other targets, but no such luck.
The SYNs of China
I started opening up the permitted traffic little by little (by manipulating subnet rules in iptables), paying attention to the half-connections. With every little opening I would see a flood of SYNs barging in and I would block the IP addresses of some of the bigger offenders. This wasn't exactly helping and it was taking too long. There were just too many IP addresses.
Curious, I decided to look up some of these IP addresses on arin.net and unbelievably all of them had been assigned to China, hundreds of subnets consisting of thousands of Chinese IPs working diligently to knock my site offline. Now it is possible that attack itself hadn't originated in China and the IP's were spoofed, but I would give that a very low probability.
Rescuing the SYN-king Server
It was time for drastic measures to save my Web site and that meant blocking China completely. I hate blocking traffic, it goes against the very spirit of the Internet but at this point I had no option. Thankfully I was able to find a site (cited below) that had a list of IP addresses assigned to China. This is a big and dynamic list and I imported the whole list into my firewall block rules and with that hashemian.com was humming again. For added measure I also hardened the TCP/IP stack on my server a bit to better withstand SYN flood attacks (sources cited below).
As mentioned, it upsets me to have blocked so many addresses on my server and in doing so also taking up server resources. My site is insignificant, but if everyone blocked everyone else, imagine what this fragmented Internet would be like. At my day job I also see a lot of similar abuse coming from China. Other places such as Russia, Nigeria, and Estonia dish out their own abuses, but this sort of heavy-handed, fatal reconnaissance and attack is almost exclusive to China. Why was my site targeted? Was it some sort of a drive-by from a robot that got stuck and kept on hammering away?
I suppose I'll never know. But I know this won't be the end of it. Botnets and attackers are a lot more far-reaching than just China. They will be back with different attacks from different angles, but that's another day and another battle.
List of IP addresses assigned to China
A little preview of TCP hardening on Linux:
# TCP SYN Flood Protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3