Hashemian Blog
Web, Finance, Technology, Running

July 3, 2017

The Long, Hard and Possibly Foolish Path to SSL/TLS Security

by robert hashemian @ 10:57 pm
Filed under: internet,web — Tags: , , , , , ,

... or TLS 1.2 on Fedora Core 14/FC14 and other older Linux versions

With the chorus of secure browsing  getting louder and becoming more prevalent,  HTTPS migration is becoming inevitable. Going secure is a pretty major undertaking, fraught with numerous pitfalls. It starts with the source files that produce the html pages and it could get ugly if there's even one element in a page that is called over http rather than https, no green padlock in that case. The Protocol-relative URL (//) instead of the hard-coded http:// or https:// is quite helpful to that end. That's one of side of the equation. The other is the server itself.

I run my site on an older server with an old Fedora Core OS (FC14) and by extension an older version of web server software, Apache 2.2.17. Over the years I have updated a few components here and there and fixed and customized a bunch of others, especially after new vulnerabilities have popped up. Updating the server to the latest and greatest version would be a non-trivial task for me. The old hardware may not be sufficiently supportive, much of the OS customizations will be lost, migrating the data and config files will be a pain and there will be downtime as well. Yet FC14 cannot support the newer and safer SSL/TLS technologies considered acceptable by today's browsers.

At my day job, I have access to resources to overcome this problem by fronting the web server with other servers running newer technologies. For example a combination of HAProxy and Varnish provides excellent web acceleration, load balancing, and SSL termination without making any updates to the core web server. No such luck for a small time operator such as myself with limited resources, so what to do?

One approach would be to only upgrade parts of the OS and Apache (httpd program) that deal with encryption but there isn't much in terms of online resources dealing with this topic other than the customary advice to upgrade the OS. In the end this became a long process of trial and error but it was a successful endeavor with a good bit of leaning as a bonus. Here's how I did it.

Apache 2.2.17 running on Fedora Core 14 can be configured for SSL, however it can only provide support for up to TLS1.0, with older cipher suites and the weak RSA key exchange. I had already patched OpenSSL after the Heartbleed bug had become public but what I needed were newer version of libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries used by mod_ssl.so, a module used by httpd to enable SSL.

I downloaded the source and built OpenSSL version 1.0.1u. Building applications from source code in Linux is usually a three-step process, configure, make, and make install. After 'make install' The new OpenSSL libraries were placed in an alternate directory, /usr/local/ssl, instead of overwriting their main system counterparts.

Next step was to incorporate the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries into mod_ssl.so and my tool of choice for that was patchelf. I downloaded and built patchelf 0.9.

Before using patchelf I took one step which I am not sure if it were necessary in hindsight. That step was adding the location of these new libraries to a conf file under /etc/ld.so.conf.d and executing the ldconfig command to add this new location to the library cache /etc/ld.so.cache used by the linker.

Here's an example of a command I used to replace one of the libraries in mod_ssl.so, I did the same for the other:

./patchelf --replace-needed libcrypto.so.1.0.0 libcrypto.so.1.0.0 mod_ssl.so

Then I used the ldd command to make sure mod_ssl.so was now linking to these new libraries.

Following an httpd restart and checking one of the secure pages I had the encouraging green padlock on Chrome browser. Indeed the page was using TLS1.2 now with a strong encryption/cipher. Yet the key exchange was using the obsoleted RSA. The new OpenSSL libraries had certainly made an improvement to mod_ssl.so but the strong key exchange element was missing.

To overcome that issue I had to rebuild mod_ssl.so with a newer version of Apache source code. That was version 2.2.32. configure was done with the following parameters:

./configure --enable-mods-shared="ssl"  --with-ssl=/usr/local/ssl

And after make I found the new mod_ssl.so in one of the subdirectories of the source files. I skipped the make install step to avoid possible complications of the new version installing itself in the system. Interestingly the new mod_ssl.so was already linked to the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries I had created in the previous step. I suppose adding that directory to the library cache had helped with that.

I placed mod_ssl.so in the folder to be loaded by /etc/httpd/conf.d/ssl.conf and restarted httpd. And it failed to start with a message about a missing symbol ap_map_http_request_error! Obviously mod_ssl.so couldn't call into this function of the older httpd (or some library) version.

To fix that error I edited the file modules/ssl/ssl_engine_io.c and replaced the line:

return ap_map_http_request_error(rv, HTTP_INTERNAL_SERVER_ERROR);

with

return HTTP_INTERNAL_SERVER_ERROR;

I admit, this is a blind alteration with possible adverse repercussions, so I don't vouch for it. Executing make once again yielded a new mod_ssl.so and this time httpd started just fine, this time with a strong key exchange added in. Testing the site with SSL Labs gave additional confirmation that SSL encryption was indeed working fine.

If wondering, I use Let's Encrypt for free SSL certificates. The recommended utility for obtaining certificates is certbot but that tool with its overly complex and finicky python virtual environment wouldn't work under FC14. The tool that worked beautifully was getssl. It's a simple and clean, yet powerful and flexible script written as one executable file in bash script. Kudos to the getssl team for creating this robust tool.

So there you have it for enabling modern SSL/TLS in an older environment, in this case FC14. The prevailing wisdom is to abandon the old OS and start off fresh with a newer platform. I don't disagree with that philosophy and I set out to do just that when I started on this journey. In the end, my way was possibly more difficult and more prone to pitfalls but ultimately it ended up being more satisfying and more instructive.

I haven't migrated the entire site to HTTPS yet, but you can click secure whoami to view and examine the first secure page.

 

November 6, 2016

HTTP to HTTPS Migration

by robert hashemian @ 10:26 pm
Filed under: google,internet,web — Tags:

https-ssl-tlsA universally secure internet may have its defenders and detractors but like it or not, Google is going to force site encryption (https) across the board.

First it was the SEO penalty threat, supposedly giving higher scores to secure sites but it doesn't seem like that worked out great. I think Google recognized that just giving prominence to secure sites over plain ones might lead to low quality sites stealing rankings from reputable ones simply by going encrypted. That would have meant poor search results pages, possibly alienating users and driving them to competitors such a Bing.

Now Google is coming at this from another angle, the Chrome browser and this one may stick. As Chrome has the biggest browser market share on the market, they can shame non-encrypted sites right from the browser rather than jeopardizing the Google search engine money machine.

Beginning January 2017 Chrome will print a timid 'Not secure' next to a plain page's URL indicating it is not encrypted. But that is just the start. The plan is to make the label bolder and more colorful with the future versions of Chrome. I suspect that at some future point Chrome may require users to jump through warning messages to show a plain page. That would be much like the cumbersome steps needed today to show a page when browsing to a secure page with a broken or invalid certificate.

The process of migration from a plain site to an encrypted site starts with obtaining a site certificate. This used to be an expensive proposition but nowadays a basic one can be had for free. In terms of the web server there are 3 ways to migrate a site from plain to secure:

1- In-place migration of the web server application - Just about any web server on the market today can handle secure connections as well as plain ones. The process generally involves installing the certificate, making some configuration changes and the site goes encrypted. Servers with multiple domains may however need an upgrade. For that, check for SNI support. For example Microsoft's IIS below version 8 does not support SNI. And if you have users that are still on Windows XP, good luck. SNI isn't supported on that platform at all.

2- Using an https appliance - Here the web server infrastructure is left intact but instead it is fronted by another server or service known as an https appliance or SSL termination. There are many such appliances on the market that are relatively easy to set up. There are also open source products such a Nginx or HAProxy that require a bit more tech know-how. In both cases they are deployed by installing the corresponding domain certificates and exposing them to the internet traffic. Internally they access the actual web server via plain http and return the page to the users encrypted over https.

3- Using a CDN - This is similar to the 2nd method, except that the appliance is actually managed by another company, like CloudFlare (free), Akamai or  CloudFront among others, in the cloud. The benefit is that little administration is required and in some  cases, like CloudFlare, even the certificate is pre-handled. The downside is giving up a certain level control and trust which a business may not be comfortable with.

Going https is not a trivial task, specially for the less tech savvy. But at least there are a number of available migration choices, each with a number of product options. They have various degrees of convenience, efficiency, and precision but eventually one must be chosen as the https migration seems inevitable. How would this site migrate to https? Remains to be seen.

... And now here's a glimpse: The Long, Hard and Possibly Foolish Path to SSL/TLS Security - TLS 1.2 on Fedora Core 14.

July 22, 2015

The Dawn of AWS Zombies

by robert hashemian @ 9:34 am
Filed under: hacking,internet — Tags: ,

awsOne of the less enviable tasks in a techie's life is identifying bogus robot traffic on their networks. Robots suck up bandwidth without giving anything in return and in most cases try to brute-force their way into systems and steal information and then assimilate their target hosts into new recruits in their army of zombie robots.

Identifying and neutralizing robots is hard enough, specially those hunting in packs causing DDoS headaches most of the time, but in past there used to be time, resources and funding barriers which moderated these attacks. With cloud services those barriers seem to have all but vanished and based on what I can see, AWS (Amazon Web Services) is one of the ugliest actors on the market. So how are AWS zombies created?

  • Hacker uses a stolen credit card to set up an account on AWS or hacks an existing AWS account.
  • Hacker spins up multiple virtual machines under this account. Or hacker breaks into a legitimate AWS virtual machine.
  • Hacker installs robot apps on one or more virtual machines and launches attacks.
  • Successful attacks bring more power to the hacker.
  • At some point AWS or the legitimate account holders notice high usages in processing, storage, and bandwidth and shut down the operation but by then the damage is done.

Could AWS be complicit in this type of activity? Perhaps not actively, but there is a passive element here as well. I'm sure they won't admit to it, but if a legit account is broken into and its cloud services are stolen, would AWS even care? They just blame the user for being careless and charge him for the usage. AWS may exercise more care in terms of blocking accounts with stolen credit cards because they may not be able to squeeze money out of those cases.

But even then, Amazon is so big with such vast resources that these cases may not even register as a blip on their radar. So the cycle of spawning AWS zombies will never cease and Amazon continues wasting our time, resources and bandwidth with impunity.

It may be an overkill to completely block AWS, but if partial blocking becomes necessary, a list of their public IP address ranges is published here and in json format here.

June 27, 2015

Craigslist eBay Motors Car Scam

by robert hashemian @ 1:09 pm
Filed under: internet,web — Tags:

ebay-craigslist-car-scamI'd been in the market for a used car when a too good price on Craigslist caught my attention. I'd sold a street bike on Craigslist a few years ago and had a good experience so figured to go into this but with raised antennas.

An email later, the seller reveals a sob story about the car belonging to her dead husband and wanting to move on. The car's in a great shape with all paperwork in order. Sounds plausible, so can I see the car? Seller replies the car is in some eBay garage across the country in lot number so and so.

No worries, she just needs my info and eBay will contact me about payment. The money will remain with eBay until I receive the car and I have 10 days to inspect it. If any issues, I can return it at no cost to me.

So I ask for the eBay page where the car is listed. Seller says she took it down because of the fees. But really, eBay will make all arrangements.

Yeah, sure man. Of course at this point the full blown scam was obvious, but should have been obvious at Craigslist. A quick Google search revealed that this unholy Craigslist-eBay alliance bait and switch is in fact very popular and a few people have been victimized, buyers and sellers.

So why this post? Just adding one more page to Google's search results to raise the warning volume slightly more.

Read this and stay vigilant. There's plenty more info on this. Just Google it.

December 22, 2014

DNS Verification Error

by robert hashemian @ 12:55 pm
Filed under: internet — Tags:

Recently it was brought to my attention that the domain name hashemian.com has a DNS error associated with it. The domain's DNS is hosted with its registrar as many registrars provide basic DNS service for free. This service generally consists of two name servers with varying degrees of restrictions to configure zones and records.

Concerned, I headed to dnscheck.pingdom.com to verify this for myself and indeed the tool does show an SOA records inconsistency error for the domain along with a couple of warnings.

dns soa error

The warnings are attributed to the fact that the name servers do not have reverse addresses (PTR records) associated with them. Having reverse addresses is not a requirement but it is recommended.

Having consistent SOA (Start Of Authority) records on all name servers is however required. Except that in this case the inconsistency reported was only due to upper and lower case differences between the records on the two name servers and that gets into a bit of an unknown territory.

According to this RFC document: Domain Name System (DNS) names are "case insensitive". That is stated right at the top of the document in the Abstract section.

Going by the RFC, Pingdom's DNS tool is incorrect in flagging the SOA records with differing letter cases as error. One can label the DNS management work at the registrar as sloppy or clumsy, but this discrepancy should at best only rise to a warning level.

December 7, 2014

Amazon Diapers To The Rescue

by robert hashemian @ 7:55 pm
Filed under: business,internet — Tags: ,

amazon diaperWhen I opened my inbox last friday, an email from Amazon greeted me with the title: "Announcing Price Reductions for AWS Data Transfer and Amazon CloudFront"

I wondered how Amazon was going to make up the difference in the face of stiff competition from other cloud vendors. Then I saw Amazon diapers in the news and all my concerns were laid to rest, genius.

Could this have been the starter line for the executive meeting at the Amazon's headquarters? "We have a revolutionary idea to counter the AWS shrinking margins and the Fire phone losses. Please bring your attention to the baby on the PowerPoint slide ..."

Revolutionary indeed 🙂

 

September 26, 2014

Linux Shellshock Bash Bug Workaround

by robert hashemian @ 12:55 pm
Filed under: computers,hacking,internet — Tags: , ,

The warnings about the shellshock bash bug are ominous and not unfounded. This is perhaps a greater risk than Heartbleed. Here are the gory details of this bug.

To test your system for this bug run the following command from the shell:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

if you see the word 'vulnerable' anywhere in the output, like below, you have the bug.

shellshock bash bug

Because bash is such a fundamental part of Linux/Unix and used in so many ways and so prevalent, it wouldn't be that difficult for malicious hackers to use this bug to penetrate a machine and do all kinds of bad things including completely take over the machine. Web sites would the most obvious target of such attacks.

Now how to fix this. New bash versions with the bug patched have become available so users can update bash and be done. But this is not as easy to do for everyone. Some people may have older, obsolete versions of Linux, so they may not find the new patched bash version. They would need to get the source code and the patches, and then build and install it themselves. Yes, I know everyone should be on the latest version of everything, and I am guilty as charged, but let's dispense with the tarring and feathering for now.

Redhat however, in its haste and panic, had released a workaround on this page with a small block of C code that once installed, would disable function definitions and therefore mitigate this risk. They called it dangerous because one must assume this workaround would disable a legitimate feature of bash and possibly cause system failure if it were being used. Unfortunately a while later this workaround vanished (Update: actually here is the Redhat page for LD_PRELOAD mitigation. I don't know,  maybe the page never vanished at all. Just use the steps on that page then), but not before I had availed myself to it. For me, the ease and speed of its deployment made it worthy of a try. And here are the steps.

1- Put the following C code in a new file, bash_ld_preload.c.

#include <sys/types.h>
#include <stdlib.h>
#include <string.h>

static void __attribute__ ((constructor)) strip_env(void);
extern char **environ;

static void strip_env() {
  char *p,*c;
  int i = 0;

  for (p = environ[i]; p!=NULL;i++ ) {
    c = strstr(p,"=() {");
    if (c != NULL) {
      *(c+2) = '\0';
    }
    p = environ[i];
  }
}

2- Compile bash_ld_preload.c to get bash_ld_preload.so using the following command.

$ gcc bash_ld_preload.c -fPIC -shared -Wl,-soname,bash_ld_preload.so.1 -o bash_ld_preload.so

3- copy bash_ld_preload.so to the /lib/ directory like so:

$ cp bash_ld_preload.so /lib/

4- Add the following to the file /etc/ld.so.preload on a line by itself:

/lib/bash_ld_preload.so

5- Restart all relevant services or just reboot the system to be sure.

 

There you have it. I deployed this on several machines that run various applications. It killed the bug and there were no adverse effects. That means that those machines were not using the function definition feature of bash. Of course at some point we may write code or install applications that need to use this feature and if we have forgotten about this workaround, there will be a lot of head-scratching.

So, use the above workaround at your own risk. It will probably work for you, but the best approach as always is to update your platform and of course your version of bash.

April 3, 2014

Amazon FireTV

by robert hashemian @ 6:10 pm
Filed under: internet,technology — Tags: ,

amazon-live-tvSo we have wireless-ready TVs, wireless-ready DVD players, Roku, Chromecast, game consoles, and who knows how many other devices attached to our TVs.

Now here comes Amazon with its amazingly innovative Fire TV that does incredible things such as, wait a second, stream videos to your TV just like any other device.

Why again do we need this device? Must be the cool logo 🙂

December 18, 2013

Was Bitcoin a Fad?

by robert hashemian @ 2:56 pm
Filed under: financial,internet — Tags:

bitcoinRemember the Million Dollar Homepage? Back then everyone thought pixel advertising was the future of web marketing. People went crazy over it, pixel sites popped up like weed, and then the whole thing faded away like it was never there.

To me that is what bitcoin is. Sure, I have a few bitcoins and I'd like to fantasize that each will be worth a million dollars some day. But let's be real, the possibility of bitcoin fading into oblivion is so much greater. Bitcoin is nothing like gold and there are 2 reasons why it'll never achieve the success some people may dream of:

1- There may be a limited number of bitcoins that can be mined but there are no limits on how many types of crypto-currency can pop up. Everyone can come up with their own version and flood the market. There are already dozens of them out there and probably thousands vying for recognition.

2- Governments will never allow bitcoin or any other type of anarchist currency gain real traction in their countries. It's just too dangerous to their existence. We've already seen moves by China and Europe to crack down on bitcoin. More will come if bitcoin's popularity survives.

The bitcoin fad will pass just like many others have before it. Something else will eventually come along and capture people's attention and what will be left of bitcoin will be http://en.wikipedia.org/wiki/Bitcoin.

July 17, 2013

Network Solutions, More Like Network Problems

by robert hashemian @ 10:16 pm
Filed under: hacking,internet — Tags: ,

Network Solutions (netsol), the company behind domain names had a rough day today and it dragged its customers down with it. Apparently a DDoS attack knocked out their network making hosted web sites and DNS servers inaccessible. This site, while not hosted on netsol, does have its name servers hosted with them and so it had several outages while netsol was combating the attack.

I don't understand how a company like netsol could fall prey to such attacks. Netsol has been around for decades, they are the original Internic, the only domain provider back when domains were free. I'm sure they have deep pockets and lots of experts working for them. Surely they have fat enough pipes to absorb such attacks and leave plenty of capacity for their users. And to make matters worse, the company's social outlets like Facebook and Twitter were silent for hours during the outage.

Things seem to be back to normal now, but if these guys can't get it right, what hope is there for the rest of us?

Older Posts »

Powered by


Read Financial Markets  |   Home  |   Blog  |   Web Tools  |   News  |   Articles  |   FAQ  |   About  |   Privacy  |   Contact
Donate Bitcoin: 1K9TzBvQ2oaEb4tX9t2vKDtZouMcpfV6QF
© 2001-2017 Robert Hashemian   Powered by Hashemian.com