Hashemian Blog
Web, Finance, Technology, Running

The Great SYN Flood of China

by @ 9:55 pm
Filed under: hacking,web — Tags: ,


I wake up yesterday morning and while still in bed I get the dreaded site-down alert from Pingdom on my smartphone. When a Web site goes down there are a number of simple preliminary steps one takes to pinpoint and fix the problem. Is the ISP having an outage? Are the modem and router up? Is the server up and is the Web service running?

The server was up but the Web service was unresponsive. The quick and dirty steps are restart the Web service, no dice there. Ok, reboot the server, still no good. It was time do drag myself over to my desk and login to the Linux server to investigate more. Going through a bunch of diagnostics steps, this is what I saw:


The Flood of The SYN-tury

Those familiar with the TCP handshake know that the session setup consists of a SYN packet from host to server, followed by a SYN-ACK packet from server to host and finally a ACK from the host to server and the connection is established. When one sees reams of SYN_RECV on the server it is indicative of a possible attack where a host or a group of them flood the server with the first SYN but they spoof their IP addresses or just snub the server's SYN-ACK packet saddling the server with these half-open connections, each one taking up a bit of the server's resources.

The server eventually cleans up the half-open connections but if the zombie connection numbers rise too fast, they exhaust the server and no more connections are accepted; the server goes offline. This is known as a SYN flood attack and it eventually leads to a condition known as DoS (Denial of Service) or the more dreaded form, DDoS (Distributed Denial of Service).

By now I was late for work, so I just blocked all traffic to the server and went to my day job. The server remained crippled all day (apologies to the users of my utilities) until I returned home and began the process of resolving the issue. I was hoping that by then the attackers had moved on to other targets, but no such luck.

The SYNs of China

I started opening up the permitted traffic little by little (by manipulating subnet rules in iptables), paying attention to the half-connections. With every little opening I would see a flood of SYNs barging in and I would block the IP addresses of some of the bigger offenders. This wasn't exactly helping and it was taking too long. There were just too many IP addresses.

Curious, I decided to look up some of these IP addresses on arin.net and unbelievably all of them had been assigned to China, hundreds of subnets consisting of thousands of Chinese IPs working diligently to knock my site offline. Now it is possible that attack itself hadn't originated in China and the IP's were spoofed, but I would give that a very low probability.

Rescuing the SYN-king Server

It was time for drastic measures to save my Web site and that meant blocking China completely. I hate blocking traffic, it goes against the very spirit of the Internet but at this point I had no option. Thankfully I was able to find a site (cited below) that had a list of IP addresses assigned to China. This is a big and dynamic list and I imported the whole list into my firewall block rules and with that hashemian.com was humming again. For added measure I also hardened the TCP/IP stack on my server a bit to better withstand SYN flood attacks (sources cited below).

As mentioned, it upsets me to have blocked so many addresses on my server and in doing so also taking up server resources. My site is insignificant, but if everyone blocked everyone else, imagine what this fragmented Internet would be like. At my day job I also see a lot of similar abuse coming from China. Other places such as Russia, Nigeria, and Estonia dish out their own abuses, but this sort of heavy-handed, fatal reconnaissance and attack is almost exclusive to China. Why was my site targeted? Was it some sort of a drive-by from a robot that got stuck and kept on hammering away?

I suppose I'll never know. But I know this won't be the end of it. Botnets and attackers are a lot more far-reaching than just China. They will be back with different attacks from different angles, but that's another day and another battle.

List of IP addresses assigned to China

Hardening TCP/IP

A little preview of TCP hardening on Linux:
# TCP SYN Flood Protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3


by @ 11:13 pm
Filed under: hacking,internet — Tags: ,

To the couple of visitors of this website, I'm sorry for the 2-day outage earlier this week. It was a DDOS (distributed denial of service) attack and I never found out who was behind it and why.

The problem started in the early morning hours with an outage alert from the remote monitoring service. The site was down and the server wasn't even responding to SSH login. Jumping directly on the server, I could already tell something was wrong by the loud sound of the fan. Indeed the load was in the 40's when it usually hovers around 0.25 and inbound traffic utilization was at saturation levels.

Realizing that I've been wrong on blaming server issues on attacks, I did what every server admin does at the first sign of trouble, reboot. No dice, the server load soon went sky-high again. So I blocked outside connections to apache and started running some simple tests to check the server health. CPU, RAM and IO checked out fine under some local test load. No, this was something else. The logs finally indicated the problem:

-- possible SYN flooding on port 80. Sending cookies.

Looking at the connections (using netstat), there were hundreds of SYN_RECV records hanging around from various IP's. Obviously the server was under a SYN flood DDOS attack. Using iptables to block the offending IP's was no help. Most likely the ip addresses were fake and combating them was like fighting a tidal wave.

The attack continued throughout the day with no relief and finally in the evening I contacted my ISP to see if they can rescue me. I didn't have much hope, but I almost lost it when the technician asked: "Huh? You have a sink flow attack? Could you spell that?" So much for tech support.

My best option was to lay low and take the abuse and hope the attacker(s) will get bored and move on. And that's exactly what they did. Almost as fast as it started, the attack stopped in the wee hours of the second day and I could finally bring the server back online.

Moral of the story, DDOS attacks are tough enough to combat for big shops. Small guys like me don't stand a chance against them. The best solution is to wait them out and hope the attacker moves on. Also small sites aren't lucrative enough to get expert support from their ISP's. The best that can be hoped for is to ask the ISP for a new set of IP's and still there's no certainly that'll stop the attackers.

As for this attacker(s) and their intent, it remains a mystery. Perhaps it was a script kiddie rolling through a bunch of victim hosts, or someone testing an attack platform or algorithm, or a mistake specifying a domain or IP in the attack vector. This site is just too small for bragging rights or boosting egos. There are much tastier targets out there for attackers to prove their expertise and flaunt their skills. Then again why use your smarts to attack sites instead of doing something constructive?


Powered by

Read Financial Markets  |   Home  |   Blog  |   Web Tools  |   News  |   Articles  |   FAQ  |   About  |   Privacy  |   Contact
Donate Bitcoin: 1GfrF49zFWfn7qHtgFxgLMihgdnVzhE361
© 2001-2021 Robert Hashemian   Powered by Hashemian.com