Hashemian Blog

Web, Finance, Technology

To the couple of visitors of this website, I'm sorry for the 2-day outage earlier this week. It was a DDOS (distributed denial of service) attack and I never found out who was behind it and why.

The problem started in the early morning hours with an outage alert from the remote monitoring service. The site was down and the server wasn't even responding to SSH login. Jumping directly on the server, I could already tell something was wrong by the loud sound of the fan. Indeed the load was in the 40's when it usually hovers around 0.25 and inbound traffic utilization was at saturation levels.

Realizing that I've been wrong on blaming server issues on attacks, I did what every server admin does at the first sign of trouble, reboot. No dice, the server load soon went sky-high again. So I blocked outside connections to apache and started running some simple tests to check the server health. CPU, RAM and IO checked out fine under some local test load. No, this was something else. The logs finally indicated the problem:

-- possible SYN flooding on port 80. Sending cookies.

Looking at the connections (using netstat), there were hundreds of SYN_RECV records hanging around from various IP's. Obviously the server was under a SYN flood DDOS attack. Using iptables to block the offending IP's was no help. Most likely the ip addresses were fake and combating them was like fighting a tidal wave.

The attack continued throughout the day with no relief and finally in the evening I contacted my ISP to see if they can rescue me. I didn't have much hope, but I almost lost it when the technician asked: "Huh? You have a sink flow attack? Could you spell that?" So much for tech support.

My best option was to lay low and take the abuse and hope the attacker(s) will get bored and move on. And that's exactly what they did. Almost as fast as it started, the attack stopped in the wee hours of the second day and I could finally bring the server back online.

Moral of the story, DDOS attacks are tough enough to combat for big shops. Small guys like me don't stand a chance against them. The best solution is to wait them out and hope the attacker moves on. Also small sites aren't lucrative enough to get expert support from their ISP's. The best that can be hoped for is to ask the ISP for a new set of IP's and still there's no certainly that'll stop the attackers.

As for this attacker(s) and their intent, it remains a mystery. Perhaps it was a script kiddie rolling through a bunch of victim hosts, or someone testing an attack platform or algorithm, or a mistake specifying a domain or IP in the attack vector. This site is just too small for bragging rights or boosting egos. There are much tastier targets out there for attackers to prove their expertise and flaunt their skills. Then again why use your smarts to attack sites instead of doing something constructive?