Hashemian Blog
Web, Finance, Technology, Running

July 22, 2021

Migrating To Cloud, Digital Ocean, Cloudflare

by robert hashemian @ 4:14 pm
Filed under: computers,email,internet,technology,web — Tags: , , , ,

When I started this site decades ago, I followed the usual path at that time to launch sites, shared hosting. There were many vendors to choose from but nothing like the quantity and diversity of what’s available today. I registered the domain, settled on a small vendor for $5/month, got my cPanel and terminal login and off I went.

My server didn’t have much processing juice, storage, and bandwidth compared to today’s levels but plenty enough to host my website and email and other services within the same account. Things were running smoothly but as you’d guess, the account eventually started to push the envelope with the hosting limitations and over the next few years I migrated from vendor to vendor in the perpetual quest for more resources and better prices.

Still all this time I was a slave to the hosting companies and their rules. If my services were abused by outsiders my account was suspended. If my site was spamvertized, my account was suspended. They’d change prices, modify account agreements, deprecate services, migrate servers, upgrade products, get merged or acquired and I had to go with the flow.

So a few years ago I finally decided to give self-hosting a try. I signed up for Comcast Business with static IP addresses, got a decent used computer and migrated all the services to a corner of my home. I was paying more for business Internet at home but I was saving on hosting costs.

Self-hosting Server at Home

This arrangement worked fine for many years but there were downsides too and with the passage of time those downsides became more prominent. To cite a few,

  • Being in a residential zone, power outages are inevitable and my puny UPS couldn’t handle long blackouts, such as those arising from intense storms.
  • Computers and equipment crash or power may be cut because of workers or a tripped breaker. If there’s no one at home to power up the equipment, services would remain offline.
  • The ambient temperature must be kept at a reasonable level. That means running the A/C on hot days even if no one is at home.
  • Hardware failures would mean outages until parts could be replaced and of course a good backup strategy is a must.
  • Software updates are necessary to support the latest or minimum required protocols or to patch vulnerabilities. For example making sure that TLS1.2 or IPv6 are supported or patch for Shellshock or Heartbleed bugs.
  • Monitoring and battling abusers and hackers become a daily stressful job.
  • With the static IP, it becomes impossible to be even slightly anonymous while surfing, unless one pays for a separate internet service.

In short, self-hosting became way too onerous and the cloud had finally reached a robust point where migrating back to providers could be a prudent move once again.

The first service to be migrated was email. I have covered the email migration to Google Workspaces in a previous post and have not looked back. The stress relief was almost immediate.

A few months later it was time to migrate the web services. For hosting I chose Digital Ocean. I had used Digital Ocean before for my day job and was impressed with their facilities and prices. Unlike the bigger players in the cloud space such as AWS or Azure that can inundate users with options, Digital Ocean has simple and straight forward options and pricing to choose from. I quickly set up a droplet (their parlance for a virtual server), transferred the files over, tested the web services, fixed the errors and incompatibilities and finally put the server into production. (My referral link, Get $100 credit with Digital Ocean.)

Shutting down the home server for the last time was a bitter sweet moment but there was no turning back and there was a tremendous feeling of relief.

The final piece of this project came with the addition of Cloudflare. Essentially Cloudflare is a web acceleration and SSL/TLS termination service and they have a lot of free services for the small operators. A few noteworthy advantages of Cloudflare are,

  • IPv6 reach. If a website is only available on IPv4, suddenly it’ll be accessible to all IPv6 clients as well.
  • Powerful and flexible firewalling and security capabilities, including DDoS handling.
  • Latest TLS and HTTP protocols.
  • Powerful web caching and acceleration features with automatic CDN.
  • Reliable and fast DNS hosting.
  • Web analytics.

A few months have now passed since I moved my server to the cloud and perhaps the only regret is not doing it sooner. I am saving money by using less power at home, terminating the static IP’s, and downgrading to lower internet speed. But more importantly I am saving my sanity by drastically reducing the stress of maintaining my own server.

Also, with the static IP’s gone, I can now replace the Comcast assigned cable modem with my own modem and hopefully save even more money every month. But that’s another project and maybe another post.

April 23, 2021

JAMstack - How To Make The Web Slower

by robert hashemian @ 10:47 am
Filed under: web — Tags: ,

Starting a few years ago everywhere you look, sites are getting updated with a new approach known as JAMstack. JAM, initials for JavaScript, API, and Markup is lauded as more revolutionary, more scalable, speedier, and more flexible than the older style website development.

Some of these claims may be true but for the most part the web crowd has drunk the Kool-Aid and has jumped on the bandwagon, lest be accused of being stuck in the 90's.

Think back about how websites used to work. User accessed a page, the server at the other end, hunted and gathered the data, formatted the page, and threw it back at the browser. It was basically a bit of a delay with a blank page and then an immediate fulfillment. Yes, the designs were sometimes ugly, the fonts were small, and there wasn't much white space but everything was there.

Look at how modern sites work today. There is a shorter initial delay, but the page only fills in with a bunch of inane placeholders. These are a collection of gray boxes, spinning bullets, or wavy bars but there's nothing in them until the browser goes out and fetches the data to fill these placeholders and at times that job seems to take an eternity.

I used to login to my banking site, click on links and the pages would almost immediately return filled with all the necessary info. Now I login and wait and wait and wait for each section to fill in while looking at a bunch of useless spinners and gray boxes.

Call me old-fashioned, and I won't deny it, but is this really called progress? Couldn't we have just cleaned up the pages with better design, more white space, and more appealing fonts using newer styling techniques, instead of creating this monstrosity?

Of course we all want to brag about how our sites are designed using the latest versions of React, Angular, Node.js, and Flask. Weren't JQuery and CSS good enough for us? Did we really need to push a ton of junk into the browser in the name of client-side rendering?

The difference between the old and new sites wasn't as stark to me until I was tasked with migrating my company's CRM platform from Goldmine to Salesforce.com. Salesforce users can operate the site in two distinct formats. The Classic format and the new JAMstack format, known as Lightning Experience which users are slowly being pushed into.

I started out with the classic version. It looks dated but it's fast and responsive. It does the job and does it quickly. But knowing that the classic version will eventually be retired I forced myself to use the lightning version. Compared to classic, lightning is slow and clunky. The pages start with a big spinner, then switch to smaller spinners and then more spinners inside of those and you wait and wait until all the data loads. Then you notice that all the data hasn't loaded because as you scroll down you are confronted with more, you guessed it, spinners. But the design is something else. There are bears dancing, tigers prancing, butterflies fluttering their wings in blue skies with puffy clouds roaming everywhere. I don't know if I'm using a CRM platform or looking at a caricatured Bob Ross painting.

Now I have nothing against cartoon characters zipping around. The lightning experience is an experience alright and admittedly the design is actually pretty nice but anything but lightning fast once you've used the classic version.

You could say that I'm nostalgic for the old days of Perl CGI, and perhaps that's true a little. But I'm actually more nostalgic for how much faster the web used to be at one time, slow modem baud rates notwithstanding. But this is the state of the web as we have today. It has been decided that JAMstack is the way to go even if it's at the speed of molasses, or should I say, JAM.

January 7, 2019

Goodbye to Pingdom

by robert hashemian @ 10:47 pm
Filed under: web — Tags:

I have used the free Pingdom service for nearly 9 years to monitor the health of this site. Over the years it has been a helpful service and at times they would add even more useful features.

Then came 2014 when SolarWinds, a public-then-private-then-public company catering to businesses with their analytics products and services, acquired Pingdom. The writing was on wall. Soon the free tier monitoring service started to lose features and it became less and less useful over time.

Today came the final nail in the coffin, Pingdom will be ending the free service next month. Can't say it was unexpected. I can understand the company's position not to bother with small-time operators such as this site who want a free service. I can only imagine the company meeting where the termination decision was made with the boss' opening kickoff line, "We've had enough of these leeches, let's cut these parasites loose."

Instead Pingdom opted for an obvious self-serving email masquerading as a customer-serving initiative. I have no expectation of these companies providing free services to users, but they should try to be a bit more honest in their messages instead of this opener, dripping with insincerity.

They could have simply said that they'd want to get paid for the services they provide and leave it at that. As I deleted my account with Pingdom today, I felt some gratitude for the years of free monitoring. Then I promptly migrated to another company, StatusCake, which provides a similar web monitoring service. Yes, it is free, at least for now.

Must admit that it wasn't a sad goodbye though. My philosophy is, never get too attached to anything to let it go.

 

June 8, 2018

The GDPR Mess

by robert hashemian @ 4:35 pm
Filed under: business,internet,law,web — Tags: ,

With GDPR (General Data Protection Regulation) being in full force since May 25, 2018, one must assume that the privacy and security of users are now fully protected. I think it’s an understatement to call that claim an over-exaggeration.

GDPR is a European regulation designed to protect the privacy of European citizens, giving them full control over their personal information. For most website operators it translates to getting users’ permission before doing anything with their data and deleting that data upon request.

While on the surface it is a well-intentioned law, little doubt remains that it has morphed into a giant confusion. Fact is no one really knows all the subtleties of this law and no one knows how to correctly implement it.

First there was a barrage of emails from companies proclaiming that they had new privacy laws, except that who has the time to click on every email and read reams of legalese nonsense.

Now we have the omnipresent ridiculous popup/slider on sites declaring some inane cookie policy for the site with a button to accept the terms. This site is guilty of that too, you might have noticed a cookie disclaimer sliding up from the bottom of the screen. The popup is just a utility script hosted on some site and I have no idea how it helps with your privacy and security while you are on this site.

Ironically, your privacy and security was just fine on this site prior to showing you the GDPR cookie notice. No data was being collected on you, no cookies were being stored on your browser, and no tracking was being done. Of course the Google services used on this site do some of those things and those are separately covered by Google’s privacy policies.

Now with the introduction of the cookie popup, this site has to use cookies to keep track of the fact that the user has been to the site and accepted the terms. In other words this site has to tell users that is uses cookie because it uses cookies to tell users that it uses cookies. And now the site hosting the popup code knows about the user too. Moreover the user that has just arrived to the site is not going to take the time to read all the cryptic nonsense in the privacy policy. Instead s/he is going to accept everything and continue. Now the site can do whatever it wants with the user’s data and it has explicit permission from the user. That provides a pretty strong incentive to abuse the data without any fear of legal consequences.

Finally, how does the European law expect a small time blogger provide the same level of privacy provisions of Amazon or Facebook to its users? Those are companies with billions of dollars at their disposal and an army of developers, attorneys and consultants.

Now comes GDPR with its esoteric rules to confound the small sites or even worse shut them down because they didn’t ask a silly question with a checkbox next to it. So much for democratizing the Internet where the small guys should have a shot at having their voice heard too.

But for now GDPR is here so by all means, read the disclaimer, visit the privacy page and click the stupid button. Don’t worry, your private data is safe with this site, especially since it doesn’t even ask for it.

September 14, 2017

A Simple Post-HTTP-to-HTTPS SEO Checklist

by robert hashemian @ 3:21 pm
Filed under: google,web — Tags: , ,

With Chrome version 62 arriving next month Google will begin making good on the promise of warning users when they land on non-secure (non-SSL, non-TLS) sites. This will be subtle at first with a light gray warning on pages that contain any input forms. This warning message will get progressively prevalent and prominent with every new version of Chrome and one must imagine other browsers will follow suit.

Another angle where HTTPS is being pushed is with AMP pages. Secure AMP pages are widely preferred by Google over the non-secure ones. In fact it seems non-secure AMP pages are not even picked up by Google News. That should give content and news sites a serious dose of inducement to go secure if they'd want better representation in the mobile world.

This blog has already covered available options to make a site secure, but once secure what can sites do to effectively promote their new secure status to search engines and by extension to their audience?

Here is checklist of steps to take once a site is migrated to the secure HTTPS/SSL.

  • Test the site with a reputable SSL/TLS utility such as https://www.ssllabs.com/ssltest/ and aim for a high grade.Make sure all pages get a green padlock. For that, all page elements' URLs must be relative or start with https:// or //. Either manually update them, use plugins for CMS's like WordPress, or use mods for servers, for example mod_substitute for apache.
  • Use header Content-Security-Policy: upgrade-insecure-requests or its meta tag equivalent. Not all browsers support this CSP header but majority do. This header instructs the browser to upgrade all HTTP elements on the page to HTTPS equivalents.
  • Use canonical headers or link tags to point to the HTTPS versions of your pages. ala, https://support.google.com/webmasters/answer/139066. The canonical tag is used to point search engines to the most desirable and valid version of a page.
  • Redirect all your HTTP pages to their HTTPS versions. What you want here is a hard or permanent redirect, also known as 301 redirect. This is best accomplished as a header response code and for that access to web server configuration is needed. There are alternative redirection methods such as using the refresh meta tag or the javascript location object, but server response header works best.
  • Use Google Search Console (previously known as Webmaster Tools) to advise Google of your site and to some extent instruct Google on crawling and indexing your pages via Sitemaps. If you already have the non-secure version of your site in Search Console that’s not enough. You must now include the HTTPS version of your site. Search Console is also a great tool for monitoring how Google interacts with your site. It even sends emails if it runs into any issues such as inability to crawl your side or finding malware.
  • If you use freebie certs, use a reputable certificate authority. For example StartSSL certificates are no longer trusted by some browsers, but Let’s Encrypt is fast gaining momentum. There are drawbacks such as lack of wildcard certificates or shorter validity durations so it takes a bit more management effort in return for no cost.
  • Utilize HTTP Strict Transport Security (HSTS) policy for your site. This policy instructs browsers to only interact with your site via HTTPS for a specified duration of time. This is strictly a response header field so access to the server configuration is necessary. It is doubtful that HSTS will improve search engine rankings, but it certainly doesn’t hurt and if a site has migrated to HTTPS, HSTS would be a wise security policy.

Like it or not, migrating to HTTPS is no longer a choice, unless one doesn’t mind being left behind. The prudent way of dealing with it is mapping out an HTTPS migration plan and once secure, taking steps to promote the new secure site.

August 18, 2017

WordPress Global Replace http to https

by robert hashemian @ 12:00 pm
Filed under: web — Tags: ,

If you are dealing with the pain of migrating your site from non-secure plain http to secure SSL/TLS https, then you are also dealing with the headache of making sure the elements on your pages such as images have https sources instead of http.

The reason is that if your pages are accessed over https but they contain http elements they are considered broken (mixed content) by just about all browsers. The point is that if a page is secure, everything in that page must be secure as well.

Different page elements are treated differently by browsers. As of now, Chrome loads non-secure images in the page, but the URL color goes grey instead of the reassuring green with the secure lockpad. Non-secure scripts are not even loaded, leading to page malfunction in many cases.

If you use WordPress, you know how much of a pain it is to mass replace all the image sources in your past posts from http to https. You can manually update posts which will take forever, update the backing MySQL database via SQL but that’s risky, or use a plugin but that may be overkill.

A quick way to handle this is to add a filter to transform all http occurrences to https on the fly. It won’t change the actual posts, only how they are sent to the browser. And it’s not 100%, but it’s good enough for most sites, including this one. And worst case, you can always remove it and you’ll be back to where you started.

As for the filter code, it’s a one-liner that you add to the functions.php file of your theme. It actually converts http URLs for image src’s to protocol-relative, and that does the job. Here it is and good luck:

add_filter('the_content',function($a){return preg_replace("#src=([\"'])http://#i","src=$1//",$a);});

July 3, 2017

The Long, Hard and Possibly Foolish Path to SSL/TLS Security

by robert hashemian @ 10:57 pm
Filed under: internet,web — Tags: , , , , , ,

... or TLS 1.2 on Fedora Core 14/FC14 and other older Linux versions

With the chorus of secure browsing  getting louder and becoming more prevalent,  HTTPS migration is becoming inevitable. Going secure is a pretty major undertaking, fraught with numerous pitfalls. It starts with the source files that produce the html pages and it could get ugly if there's even one element in a page that is called over http rather than https, no green padlock in that case. The Protocol-relative URL (//) instead of the hard-coded http:// or https:// is quite helpful to that end. That's one of side of the equation. The other is the server itself.

I run my site on an older server with an old Fedora Core OS (FC14) and by extension an older version of web server software, Apache 2.2.17. Over the years I have updated a few components here and there and fixed and customized a bunch of others, especially after new vulnerabilities have popped up. Updating the server to the latest and greatest version would be a non-trivial task for me. The old hardware may not be sufficiently supportive, much of the OS customizations will be lost, migrating the data and config files will be a pain and there will be downtime as well. Yet FC14 cannot support the newer and safer SSL/TLS technologies considered acceptable by today's browsers.

At my day job, I have access to resources to overcome this problem by fronting the web server with other servers running newer technologies. For example a combination of HAProxy and Varnish provides excellent web acceleration, load balancing, and SSL termination without making any updates to the core web server. No such luck for a small time operator such as myself with limited resources, so what to do?

One approach would be to only upgrade parts of the OS and Apache (httpd program) that deal with encryption but there isn't much in terms of online resources dealing with this topic other than the customary advice to upgrade the OS. In the end this became a long process of trial and error but it was a successful endeavor with a good bit of leaning as a bonus. Here's how I did it.

Apache 2.2.17 running on Fedora Core 14 can be configured for SSL, however it can only provide support for up to TLS1.0, with older cipher suites and the weak RSA key exchange. I had already patched OpenSSL after the Heartbleed bug had become public but what I needed were newer version of libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries used by mod_ssl.so, a module used by httpd to enable SSL.

I downloaded the source and built OpenSSL version 1.0.1u. Building applications from source code in Linux is usually a three-step process, configure, make, and make install. After 'make install' The new OpenSSL libraries were placed in an alternate directory, /usr/local/ssl, instead of overwriting their main system counterparts.

Next step was to incorporate the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries into mod_ssl.so and my tool of choice for that was patchelf. I downloaded and built patchelf 0.9.

Before using patchelf I took one step which I am not sure if it were necessary in hindsight. That step was adding the location of these new libraries to a conf file under /etc/ld.so.conf.d and executing the ldconfig command to add this new location to the library cache /etc/ld.so.cache used by the linker.

Here's an example of a command I used to replace one of the libraries in mod_ssl.so, I did the same for the other:

./patchelf --replace-needed libcrypto.so.1.0.0 libcrypto.so.1.0.0 mod_ssl.so

Then I used the ldd command to make sure mod_ssl.so was now linking to these new libraries.

Following an httpd restart and checking one of the secure pages I had the encouraging green padlock on Chrome browser. Indeed the page was using TLS1.2 now with a strong encryption/cipher. Yet the key exchange was using the obsoleted RSA. The new OpenSSL libraries had certainly made an improvement to mod_ssl.so but the strong key exchange element was missing.

To overcome that issue I had to rebuild mod_ssl.so with a newer version of Apache source code. That was version 2.2.32. configure was done with the following parameters:

./configure --enable-mods-shared="ssl"  --with-ssl=/usr/local/ssl

And after make I found the new mod_ssl.so in one of the subdirectories of the source files. I skipped the make install step to avoid possible complications of the new version installing itself in the system. Interestingly the new mod_ssl.so was already linked to the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries I had created in the previous step. I suppose adding that directory to the library cache had helped with that.

I placed mod_ssl.so in the folder to be loaded by /etc/httpd/conf.d/ssl.conf and restarted httpd. And it failed to start with a message about a missing symbol ap_map_http_request_error! Obviously mod_ssl.so couldn't call into this function of the older httpd (or some library) version.

To fix that error I edited the file modules/ssl/ssl_engine_io.c and replaced the line:

return ap_map_http_request_error(rv, HTTP_INTERNAL_SERVER_ERROR);

with

return HTTP_INTERNAL_SERVER_ERROR;

I admit, this is a blind alteration with possible adverse repercussions, so I don't vouch for it. Executing make once again yielded a new mod_ssl.so and this time httpd started just fine, this time with a strong key exchange added in. Testing the site with SSL Labs gave additional confirmation that SSL encryption was indeed working fine.

If wondering, I use Let's Encrypt for free SSL certificates. The recommended utility for obtaining certificates is certbot but that tool with its overly complex and finicky python virtual environment wouldn't work under FC14. The tool that worked beautifully was getssl. It's a simple and clean, yet powerful and flexible script written as one executable file in bash script. Kudos to the getssl team for creating this robust tool.

So there you have it for enabling modern SSL/TLS in an older environment, in this case FC14. The prevailing wisdom is to abandon the old OS and start off fresh with a newer platform. I don't disagree with that philosophy and I set out to do just that when I started on this journey. In the end, my way was possibly more difficult and more prone to pitfalls but ultimately it ended up being more satisfying and more instructive.

I haven't migrated the entire site to HTTPS yet, but you can click secure whoami to view and examine the first secure page.

 

November 6, 2016

HTTP to HTTPS Migration

by robert hashemian @ 10:26 pm
Filed under: google,internet,web — Tags:

https-ssl-tlsA universally secure internet may have its defenders and detractors but like it or not, Google is going to force site encryption (https) across the board.

First it was the SEO penalty threat, supposedly giving higher scores to secure sites but it doesn't seem like that worked out great. I think Google recognized that just giving prominence to secure sites over plain ones might lead to low quality sites stealing rankings from reputable ones simply by going encrypted. That would have meant poor search results pages, possibly alienating users and driving them to competitors such a Bing.

Now Google is coming at this from another angle, the Chrome browser and this one may stick. As Chrome has the biggest browser market share on the market, they can shame non-encrypted sites right from the browser rather than jeopardizing the Google search engine money machine.

Beginning January 2017 Chrome will print a timid 'Not secure' next to a plain page's URL indicating it is not encrypted. But that is just the start. The plan is to make the label bolder and more colorful with the future versions of Chrome. I suspect that at some future point Chrome may require users to jump through warning messages to show a plain page. That would be much like the cumbersome steps needed today to show a page when browsing to a secure page with a broken or invalid certificate.

The process of migration from a plain site to an encrypted site starts with obtaining a site certificate. This used to be an expensive proposition but nowadays a basic one can be had for free. In terms of the web server there are 3 ways to migrate a site from plain to secure:

1- In-place migration of the web server application - Just about any web server on the market today can handle secure connections as well as plain ones. The process generally involves installing the certificate, making some configuration changes and the site goes encrypted. Servers with multiple domains may however need an upgrade. For that, check for SNI support. For example Microsoft's IIS below version 8 does not support SNI. And if you have users that are still on Windows XP, good luck. SNI isn't supported on that platform at all.

2- Using an https appliance - Here the web server infrastructure is left intact but instead it is fronted by another server or service known as an https appliance or SSL termination. There are many such appliances on the market that are relatively easy to set up. There are also open source products such a Nginx or HAProxy that require a bit more tech know-how. In both cases they are deployed by installing the corresponding domain certificates and exposing them to the internet traffic. Internally they access the actual web server via plain http and return the page to the users encrypted over https.

3- Using a CDN - This is similar to the 2nd method, except that the appliance is actually managed by another company, like CloudFlare (free), Akamai or  CloudFront among others, in the cloud. The benefit is that little administration is required and in some  cases, like CloudFlare, even the certificate is pre-handled. The downside is giving up a certain level control and trust which a business may not be comfortable with.

Going https is not a trivial task, specially for the less tech savvy. But at least there are a number of available migration choices, each with a number of product options. They have various degrees of convenience, efficiency, and precision but eventually one must be chosen as the https migration seems inevitable. How would this site migrate to https? Remains to be seen.

... And now here's a glimpse: The Long, Hard and Possibly Foolish Path to SSL/TLS Security - TLS 1.2 on Fedora Core 14.

September 25, 2016

Online Ad to Block Online Ad

by robert hashemian @ 9:33 pm
Filed under: web —

An online ad on YouTube for an application to block online ads. Oh the irony 🙂

online ad for no online ad

April 29, 2016

The Great SYN Flood of China

by robert hashemian @ 9:55 pm
Filed under: hacking,web — Tags: ,

china-syn-flood

I wake up yesterday morning and while still in bed I get the dreaded site-down alert from Pingdom on my smartphone. When a Web site goes down there are a number of simple preliminary steps one takes to pinpoint and fix the problem. Is the ISP having an outage? Are the modem and router up? Is the server up and is the Web service running?

The server was up but the Web service was unresponsive. The quick and dirty steps are restart the Web service, no dice there. Ok, reboot the server, still no good. It was time do drag myself over to my desk and login to the Linux server to investigate more. Going through a bunch of diagnostics steps, this is what I saw:

syn_recv

The Flood of The SYN-tury

Those familiar with the TCP handshake know that the session setup consists of a SYN packet from host to server, followed by a SYN-ACK packet from server to host and finally a ACK from the host to server and the connection is established. When one sees reams of SYN_RECV on the server it is indicative of a possible attack where a host or a group of them flood the server with the first SYN but they spoof their IP addresses or just snub the server's SYN-ACK packet saddling the server with these half-open connections, each one taking up a bit of the server's resources.

The server eventually cleans up the half-open connections but if the zombie connection numbers rise too fast, they exhaust the server and no more connections are accepted; the server goes offline. This is known as a SYN flood attack and it eventually leads to a condition known as DoS (Denial of Service) or the more dreaded form, DDoS (Distributed Denial of Service).

By now I was late for work, so I just blocked all traffic to the server and went to my day job. The server remained crippled all day (apologies to the users of my utilities) until I returned home and began the process of resolving the issue. I was hoping that by then the attackers had moved on to other targets, but no such luck.

The SYNs of China

I started opening up the permitted traffic little by little (by manipulating subnet rules in iptables), paying attention to the half-connections. With every little opening I would see a flood of SYNs barging in and I would block the IP addresses of some of the bigger offenders. This wasn't exactly helping and it was taking too long. There were just too many IP addresses.

Curious, I decided to look up some of these IP addresses on arin.net and unbelievably all of them had been assigned to China, hundreds of subnets consisting of thousands of Chinese IPs working diligently to knock my site offline. Now it is possible that attack itself hadn't originated in China and the IP's were spoofed, but I would give that a very low probability.

Rescuing the SYN-king Server

It was time for drastic measures to save my Web site and that meant blocking China completely. I hate blocking traffic, it goes against the very spirit of the Internet but at this point I had no option. Thankfully I was able to find a site (cited below) that had a list of IP addresses assigned to China. This is a big and dynamic list and I imported the whole list into my firewall block rules and with that hashemian.com was humming again. For added measure I also hardened the TCP/IP stack on my server a bit to better withstand SYN flood attacks (sources cited below).

As mentioned, it upsets me to have blocked so many addresses on my server and in doing so also taking up server resources. My site is insignificant, but if everyone blocked everyone else, imagine what this fragmented Internet would be like. At my day job I also see a lot of similar abuse coming from China. Other places such as Russia, Nigeria, and Estonia dish out their own abuses, but this sort of heavy-handed, fatal reconnaissance and attack is almost exclusive to China. Why was my site targeted? Was it some sort of a drive-by from a robot that got stuck and kept on hammering away?

I suppose I'll never know. But I know this won't be the end of it. Botnets and attackers are a lot more far-reaching than just China. They will be back with different attacks from different angles, but that's another day and another battle.

List of IP addresses assigned to China
http://www.okean.com/antispam/iptables/rc.firewall.china

Hardening TCP/IP
http://www.symantec.com/connect/articles/hardening-tcpip-stack-syn-attacks
https://www.ndchost.com/wiki/server-administration/hardening-tcpip-syn-flood

A little preview of TCP hardening on Linux:
# TCP SYN Flood Protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3

Older Posts »

Powered by


Read Financial Markets  |   Home  |   Blog  |   Web Tools  |   News  |   Articles  |   FAQ  |   About  |   Privacy  |   Contact
Donate Bitcoin: 1GfrF49zFWfn7qHtgFxgLMihgdnVzhE361
paypal.me/rhashemian
© 2001-2021 Robert Hashemian   Powered by Hashemian.com