Hashemian Blog
Web, Finance, Technology, Running

April 29, 2016

The Great SYN Flood of China

by robert hashemian @ 9:55 pm
Filed under: hacking,web — Tags: ,


I wake up yesterday morning and while still in bed I get the dreaded site-down alert from Pingdom on my smartphone. When a Web site goes down there are a number of simple preliminary steps one takes to pinpoint and fix the problem. Is the ISP having an outage? Are the modem and router up? Is the server up and is the Web service running?

The server was up but the Web service was unresponsive. The quick and dirty steps are restart the Web service, no dice there. Ok, reboot the server, still no good. It was time do drag myself over to my desk and login to the Linux server to investigate more. Going through a bunch of diagnostics steps, this is what I saw:


The Flood of The SYN-tury

Those familiar with the TCP handshake know that the session setup consists of a SYN packet from host to server, followed by a SYN-ACK packet from server to host and finally a ACK from the host to server and the connection is established. When one sees reams of SYN_RECV on the server it is indicative of a possible attack where a host or a group of them flood the server with the first SYN but they spoof their IP addresses or just snub the server's SYN-ACK packet saddling the server with these half-open connections, each one taking up a bit of the server's resources.

The server eventually cleans up the half-open connections but if the zombie connection numbers rise too fast, they exhaust the server and no more connections are accepted; the server goes offline. This is known as a SYN flood attack and it eventually leads to a condition known as DoS (Denial of Service) or the more dreaded form, DDoS (Distributed Denial of Service).

By now I was late for work, so I just blocked all traffic to the server and went to my day job. The server remained crippled all day (apologies to the users of my utilities) until I returned home and began the process of resolving the issue. I was hoping that by then the attackers had moved on to other targets, but no such luck.

The SYNs of China

I started opening up the permitted traffic little by little (by manipulating subnet rules in iptables), paying attention to the half-connections. With every little opening I would see a flood of SYNs barging in and I would block the IP addresses of some of the bigger offenders. This wasn't exactly helping and it was taking too long. There were just too many IP addresses.

Curious, I decided to look up some of these IP addresses on arin.net and unbelievably all of them had been assigned to China, hundreds of subnets consisting of thousands of Chinese IPs working diligently to knock my site offline. Now it is possible that attack itself hadn't originated in China and the IP's were spoofed, but I would give that a very low probability.

Rescuing the SYN-king Server

It was time for drastic measures to save my Web site and that meant blocking China completely. I hate blocking traffic, it goes against the very spirit of the Internet but at this point I had no option. Thankfully I was able to find a site (cited below) that had a list of IP addresses assigned to China. This is a big and dynamic list and I imported the whole list into my firewall block rules and with that hashemian.com was humming again. For added measure I also hardened the TCP/IP stack on my server a bit to better withstand SYN flood attacks (sources cited below).

As mentioned, it upsets me to have blocked so many addresses on my server and in doing so also taking up server resources. My site is insignificant, but if everyone blocked everyone else, imagine what this fragmented Internet would be like. At my day job I also see a lot of similar abuse coming from China. Other places such as Russia, Nigeria, and Estonia dish out their own abuses, but this sort of heavy-handed, fatal reconnaissance and attack is almost exclusive to China. Why was my site targeted? Was it some sort of a drive-by from a robot that got stuck and kept on hammering away?

I suppose I'll never know. But I know this won't be the end of it. Botnets and attackers are a lot more far-reaching than just China. They will be back with different attacks from different angles, but that's another day and another battle.

List of IP addresses assigned to China

Hardening TCP/IP

A little preview of TCP hardening on Linux:
# TCP SYN Flood Protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3

July 22, 2015

The Dawn of AWS Zombies

by robert hashemian @ 9:34 am
Filed under: hacking,internet — Tags: ,

awsOne of the less enviable tasks in a techie's life is identifying bogus robot traffic on their networks. Robots suck up bandwidth without giving anything in return and in most cases try to brute-force their way into systems and steal information and then assimilate their target hosts into new recruits in their army of zombie robots.

Identifying and neutralizing robots is hard enough, specially those hunting in packs causing DDoS headaches most of the time, but in past there used to be time, resources and funding barriers which moderated these attacks. With cloud services those barriers seem to have all but vanished and based on what I can see, AWS (Amazon Web Services) is one of the ugliest actors on the market. So how are AWS zombies created?

  • Hacker uses a stolen credit card to set up an account on AWS or hacks an existing AWS account.
  • Hacker spins up multiple virtual machines under this account. Or hacker breaks into a legitimate AWS virtual machine.
  • Hacker installs robot apps on one or more virtual machines and launches attacks.
  • Successful attacks bring more power to the hacker.
  • At some point AWS or the legitimate account holders notice high usages in processing, storage, and bandwidth and shut down the operation but by then the damage is done.

Could AWS be complicit in this type of activity? Perhaps not actively, but there is a passive element here as well. I'm sure they won't admit to it, but if a legit account is broken into and its cloud services are stolen, would AWS even care? They just blame the user for being careless and charge him for the usage. AWS may exercise more care in terms of blocking accounts with stolen credit cards because they may not be able to squeeze money out of those cases.

But even then, Amazon is so big with such vast resources that these cases may not even register as a blip on their radar. So the cycle of spawning AWS zombies will never cease and Amazon continues wasting our time, resources and bandwidth with impunity.

It may be an overkill to completely block AWS, but if partial blocking becomes necessary, a list of their public IP address ranges is published here and in json format here.

February 26, 2015

The SSL Safety Myth

by robert hashemian @ 6:21 pm
Filed under: hacking — Tags: ,

The past week the security universe has been pounded by a whirlwind of bad press and bad actors. It all started with the news of Lenovo pre-installing adware (better yet, crapware) on new machines that would allow ads from a company with the ironic and unfortunate name, Superfish, to display context ads even when users are viewing secure web pages. The details are technical, but suffice it to say that they employ SSL certificate trickery to fool browsers and silence any possible warnings to users.

Suddenly the previously esoteric or arcane man-in-the-middle (MITM) terminology has been thrust into the mainstream and now MITM is just as well known as Ebola, even if most people have no idea how it works.

MITM - Courtesy owasp.org

MITM - Courtesy owasp.org

The bigger question however is, does ubiquitous SSL (nowadays, TLS) really make computing safer? There has been a concerted push as of late to encrypt the entire web. Google for example has suggested that it favors secure sites over regular ones.

But as evident, SSL is no panacea for security or data privacy. It does make the job of corporate security teams harder, sucks more power from infrastructures, complicates interoperability, but worst of all, gives a false sense of security to users and admins. For example, people may simply assume that with SSL they can't get infected or their private data can't be hacked.

I am obviously not against cyber security, but there are proper times and places for that. Just because something is good when appropriately applied, it doesn't mean it's good for everything all the time. Unfortunately, society always seems to over-simplify things and take everything to absurd levels using the logic, if a little is good, a lot is better.

February 17, 2015

What The Linux Ghost Bug Teaches

by robert hashemian @ 6:07 pm
Filed under: computers,hacking — Tags:

A couple of weeks ago it was revealed that a known Linux bug, Ghost (short-ish for the gethostbyname() function in the older glibc library versions) is riskier than previously thought. So the internet became abuzz with warnings to those who might not have updated their Linux distros.

I have several versions of Fedora running on various machines and updating them was simply not an option. Unfortunately they are also too old and patches are no longer available. But here comes the beauty of Linux, the open source code model. Combine that with a virtual server like Hyper-V and you have all the tools you need to create the patch yourself.

This is what I did to create patches for one of my platforms:

  • Created a guest virtual machine on the virtual server.
  • Downloaded the needed version of Fedora from this archive.
  • Installed the OS on the guest machine.
  • Downloaded the appropriate source code version of glibc. rpmfind.net is a good place to find many source code packages.
  • After installing all tools and libraries necessary to compile and build glibc, I used this StackExchange post as a guide to patch the C files based on the documented modifications and built the rpm package.
  • After installing and testing the newly built glibc library on the guest machine, I copied the rpm files to the production machine and installed them.
  • After a reboot, the bug was patched.

C code

Now many would object to running an older and unsupported version of Linux for production but I am not so sure that jumping to every new version as soon it is released contributes to additional safety. Staying with older versions does make the job of patching these sorts of bugs more cumbersome, but there's something to be said about the educational value of patching these bugs at more basic levels than just running the yum or apt-get commands. I, for one, learned quite a bit from this exercise.


September 26, 2014

Linux Shellshock Bash Bug Workaround

by robert hashemian @ 12:55 pm
Filed under: computers,hacking,internet — Tags: , ,

The warnings about the shellshock bash bug are ominous and not unfounded. This is perhaps a greater risk than Heartbleed. Here are the gory details of this bug.

To test your system for this bug run the following command from the shell:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

if you see the word 'vulnerable' anywhere in the output, like below, you have the bug.

shellshock bash bug

Because bash is such a fundamental part of Linux/Unix and used in so many ways and so prevalent, it wouldn't be that difficult for malicious hackers to use this bug to penetrate a machine and do all kinds of bad things including completely take over the machine. Web sites would the most obvious target of such attacks.

Now how to fix this. New bash versions with the bug patched have become available so users can update bash and be done. But this is not as easy to do for everyone. Some people may have older, obsolete versions of Linux, so they may not find the new patched bash version. They would need to get the source code and the patches, and then build and install it themselves. Yes, I know everyone should be on the latest version of everything, and I am guilty as charged, but let's dispense with the tarring and feathering for now.

Redhat however, in its haste and panic, had released a workaround on this page with a small block of C code that once installed, would disable function definitions and therefore mitigate this risk. They called it dangerous because one must assume this workaround would disable a legitimate feature of bash and possibly cause system failure if it were being used. Unfortunately a while later this workaround vanished (Update: actually here is the Redhat page for LD_PRELOAD mitigation. I don't know,  maybe the page never vanished at all. Just use the steps on that page then), but not before I had availed myself to it. For me, the ease and speed of its deployment made it worthy of a try. And here are the steps.

1- Put the following C code in a new file, bash_ld_preload.c.

#include <sys/types.h>
#include <stdlib.h>
#include <string.h>

static void __attribute__ ((constructor)) strip_env(void);
extern char **environ;

static void strip_env() {
  char *p,*c;
  int i = 0;

  for (p = environ[i]; p!=NULL;i++ ) {
    c = strstr(p,"=() {");
    if (c != NULL) {
      *(c+2) = '\0';
    p = environ[i];

2- Compile bash_ld_preload.c to get bash_ld_preload.so using the following command.

$ gcc bash_ld_preload.c -fPIC -shared -Wl,-soname,bash_ld_preload.so.1 -o bash_ld_preload.so

3- copy bash_ld_preload.so to the /lib/ directory like so:

$ cp bash_ld_preload.so /lib/

4- Add the following to the file /etc/ld.so.preload on a line by itself:


5- Restart all relevant services or just reboot the system to be sure.


There you have it. I deployed this on several machines that run various applications. It killed the bug and there were no adverse effects. That means that those machines were not using the function definition feature of bash. Of course at some point we may write code or install applications that need to use this feature and if we have forgotten about this workaround, there will be a lot of head-scratching.

So, use the above workaround at your own risk. It will probably work for you, but the best approach as always is to update your platform and of course your version of bash.

September 22, 2014

Bait and Switch Google Adwords

by robert hashemian @ 12:37 pm
Filed under: google,hacking,web — Tags: ,

We're all familiar with targeted banners these days. Visit a shoe site and suddenly all banners in various web sites are shoe-related.

It seems the banner scammers/hijackers have figured this out too. Recently I noticed suspicious Adwords banners originating from a site called adnxs.com.

My guess is that the malware authors use Adwords or similar networks or sub-networks to target users with certain keywords, for example shoes. They may upload legitimate ads in the beginning and may even run them for a while to gain the network's trust. But then the switch happens and malware ads such as below are displayed.

malware banner

To a lay user, a banner such as above may look legitimate enough to click which will inevitably lead to a malware download and it's game-over for that user. The banner obviously has the tell tale look of being a scam, with the "importent" update it purports to install.

Hard to say if adnxs.com or similar sub-networks are in on the scam or just look the other way as long as the money keeps coming. Whatever the case, browsers and anti-virus programs seem unable to stop these annoying and harmful banners.

October 27, 2013

Wordpress 'page_option' Hack

by robert hashemian @ 1:21 pm
Filed under: hacking,web — Tags:

wordpress-hackWordpress is a great publishing product but its popularity is also its Achilles heel. It's notorious for being a favorite target of hackers and many have been successful in compromising plenty of installations out there, including this one.

Having automated monitoring software is certainly a prudent way to stay on top of things, but in the end vigilance and bit of common sense seems to be a good way of detecting and removing attacks. Thwarting them is of course another story.

Staying with the vigilance theme, for some time I had noticed that this blog was very slow. I just attributed it to the server load or bandwidth issues but like everything else after a prolonged time of sluggish performance I turned my attention to the installation itself.

That's when I discovered the 'page_option' hack. The 'functions.php' file in my theme folder had been appended with a block calling the 'add-action' with the 'wp_head' parameter. The second parameter was from a deserialized array coming from a newly added row in the 'wp_options' table (in MySQL) with the 'option_name' field set to 'page_option'. The whole thing smelled of a hack, you know the mysterious call to decode and slice up some long encoded string. Why do hackers waste so much time with these idiotic obscurity schemes? Just dump the damn payload in. the layman won't see it and the rest can spot it from miles away, totally pointless.

A Google search brought up this reference, and the blogger's experience was very similar to mine and indeed I found the offending '/wp-includes/page.php' file just as he had. He has very good tips and hints on dealing with this hack, so head on over and give it a read.

As for me, I removed the offending block from the 'functions.php' file, delete the '/wp-includes/page.php' file, deleted the 'page_option' row from the 'wp_options' table and removed all unused themes and plugins, in case those were the hacker's conduit.

The page load times are now back to normal and for good measure I updated Wordpress to the latest version, always a wise move as they always plug new security holes.

Stay vigilant…

July 17, 2013

Network Solutions, More Like Network Problems

by robert hashemian @ 10:16 pm
Filed under: hacking,internet — Tags: ,

Network Solutions (netsol), the company behind domain names had a rough day today and it dragged its customers down with it. Apparently a DDoS attack knocked out their network making hosted web sites and DNS servers inaccessible. This site, while not hosted on netsol, does have its name servers hosted with them and so it had several outages while netsol was combating the attack.

I don't understand how a company like netsol could fall prey to such attacks. Netsol has been around for decades, they are the original Internic, the only domain provider back when domains were free. I'm sure they have deep pockets and lots of experts working for them. Surely they have fat enough pipes to absorb such attacks and leave plenty of capacity for their users. And to make matters worse, the company's social outlets like Facebook and Twitter were silent for hours during the outage.

Things seem to be back to normal now, but if these guys can't get it right, what hope is there for the rest of us?

June 10, 2013

PRISM, Spies, and Leaks

by robert hashemian @ 12:22 pm
Filed under: hacking,law — Tags: , ,

While the US and much of the world is embroiled in the so-called scandal of US government spying on its citizens, I am left wondering why this is news at all?

The Patriot Act enacted after September 11, 2001 was designed to do exactly that, spy on voice and data communications in the US. Yes there are some subtleties in the act to supposedly protect the constitutional rights of the Americans, but really, who would believe any of that?

I fail to see the uproar on the so-called leak because it's just absurd. What do people think organizations such as the NSA or CIA do all day? They listen to conversations, intercept emails, mine data, and decipher intelligence based on events. That's what they were set up to do.

When I make a phone call to my mother in Iran, the whole time there are pops, hisses, clicks, and various tones. Some may be line issues, but I bet most are a bunch of people or recording devices eavesdropping on the phone call in Iran and the US. Of course in this case, it's a bunch of boring news about family members and who got married or divorced or had a child or passed away.

So Guardian pays some low-level NSA employee in Hong Kong a bunch of money to reveal secrets about the US government spying on its people. Who knows, maybe the Chinese want to divert attention from their own hacking, or perhaps it's just a sham by the US government to serve its own end in some manner.

Whatever the case those secrets were not much of a secret at all. What's next, Iranian government spying on its people? I had no idea, that is shockingly outrageous.

November 30, 2012

Disabling SELinux

by robert hashemian @ 6:16 pm
Filed under: computers,hacking — Tags: ,

I know it's sacrilegious for some to disable a security feature on a platform, but SELinux (an enhanced Linux security feature) has left me no choice but doing exactly that on Linux.

SELinux was added to Linux to give it additional security measures beyond what it inherited From Unix. By default many of the Linux distros such as Fedora have SELinux built into their kernels and enabled upon install.

The issue is that SELinux can be so restricting and obsessive about curbing malicious activity that it can also hinder normal operations leading to server stress or errors. Having been bitten by SELinux multiple times, I have vowed to deactivate it every time I install Linux on a host. The one time I forgot to disable it, the Varnish server I have setup for my company nearly died taking the company's web site along for the ride. Looking inside the messages file, this arcane message is what I saw in prodigious numbers:

setroubleshoot: SELinux is preventing irqbalance from mmap_zero access on the memprotect Unknown. For complete SELinux messages. run sealert -l efce…

I know the security sticklers would accuse me of not setting up SELinux correctly and for the record SELinux is very configurable. But my most favorite setting for SELinux is disabling it in the /etc/selinux/config file by setting SELINUX=disabled.

I don't have the time nor the inclination to learn SELinux's every minutia, which may or may not protect my hosts completely anyways. The old fashion file permissions, file ownership, suexec, sudo, suid, running daemons with least privilege, and a good dose of firewalling is good enough for me. Feel free to disagree.

Older Posts »

Powered by

Read Financial Markets  |   Home  |   Blog  |   Web Tools  |   News  |   Articles  |   FAQ  |   About  |   Contact
Bitcoin: 1K9TzBvQ2oaEb4tX9t2vKDtZouMcpfV6QF
© 2001-2017 Robert Hashemian   Powered by Hashemian.com