Hashemian Blog
Web, Finance, Technology, Running

The Long, Hard and Possibly Foolish Path to SSL/TLS Security

by @ 10:57 pm
Filed under: internet,web — Tags: , , , , , ,

... or TLS 1.2 on Fedora Core 14/FC14 and other older Linux versions

With the chorus of secure browsing  getting louder and becoming more prevalent,  HTTPS migration is becoming inevitable. Going secure is a pretty major undertaking, fraught with numerous pitfalls. It starts with the source files that produce the html pages and it could get ugly if there's even one element in a page that is called over http rather than https, no green padlock in that case. The Protocol-relative URL (//) instead of the hard-coded http:// or https:// is quite helpful to that end. That's one of side of the equation. The other is the server itself.

I run my site on an older server with an old Fedora Core OS (FC14) and by extension an older version of web server software, Apache 2.2.17. Over the years I have updated a few components here and there and fixed and customized a bunch of others, especially after new vulnerabilities have popped up. Updating the server to the latest and greatest version would be a non-trivial task for me. The old hardware may not be sufficiently supportive, much of the OS customizations will be lost, migrating the data and config files will be a pain and there will be downtime as well. Yet FC14 cannot support the newer and safer SSL/TLS technologies considered acceptable by today's browsers.

At my day job, I have access to resources to overcome this problem by fronting the web server with other servers running newer technologies. For example a combination of HAProxy and Varnish provides excellent web acceleration, load balancing, and SSL termination without making any updates to the core web server. No such luck for a small time operator such as myself with limited resources, so what to do?

One approach would be to only upgrade parts of the OS and Apache (httpd program) that deal with encryption but there isn't much in terms of online resources dealing with this topic other than the customary advice to upgrade the OS. In the end this became a long process of trial and error but it was a successful endeavor with a good bit of leaning as a bonus. Here's how I did it.

Apache 2.2.17 running on Fedora Core 14 can be configured for SSL, however it can only provide support for up to TLS1.0, with older cipher suites and the weak RSA key exchange. I had already patched OpenSSL after the Heartbleed bug had become public but what I needed were newer version of libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries used by mod_ssl.so, a module used by httpd to enable SSL.

I downloaded the source and built OpenSSL version 1.0.1u. Building applications from source code in Linux is usually a three-step process, configure, make, and make install. After 'make install' The new OpenSSL libraries were placed in an alternate directory, /usr/local/ssl, instead of overwriting their main system counterparts.

Next step was to incorporate the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries into mod_ssl.so and my tool of choice for that was patchelf. I downloaded and built patchelf 0.9.

Before using patchelf I took one step which I am not sure if it were necessary in hindsight. That step was adding the location of these new libraries to a conf file under /etc/ld.so.conf.d and executing the ldconfig command to add this new location to the library cache /etc/ld.so.cache used by the linker.

Here's an example of a command I used to replace one of the libraries in mod_ssl.so, I did the same for the other:

./patchelf --replace-needed libcrypto.so.1.0.0 libcrypto.so.1.0.0 mod_ssl.so

Then I used the ldd command to make sure mod_ssl.so was now linking to these new libraries.

Following an httpd restart and checking one of the secure pages I had the encouraging green padlock on Chrome browser. Indeed the page was using TLS1.2 now with a strong encryption/cipher. Yet the key exchange was using the obsoleted RSA. The new OpenSSL libraries had certainly made an improvement to mod_ssl.so but the strong key exchange element was missing.

To overcome that issue I had to rebuild mod_ssl.so with a newer version of Apache source code. That was version 2.2.32. configure was done with the following parameters:

./configure --enable-mods-shared="ssl"  --with-ssl=/usr/local/ssl

And after make I found the new mod_ssl.so in one of the subdirectories of the source files. I skipped the make install step to avoid possible complications of the new version installing itself in the system. Interestingly the new mod_ssl.so was already linked to the new libcrypto.so.1.0.0 and libssl.so.1.0.0 libraries I had created in the previous step. I suppose adding that directory to the library cache had helped with that.

I placed mod_ssl.so in the folder to be loaded by /etc/httpd/conf.d/ssl.conf and restarted httpd. And it failed to start with a message about a missing symbol ap_map_http_request_error! Obviously mod_ssl.so couldn't call into this function of the older httpd (or some library) version.

To fix that error I edited the file modules/ssl/ssl_engine_io.c and replaced the line:

return ap_map_http_request_error(rv, HTTP_INTERNAL_SERVER_ERROR);

with

return HTTP_INTERNAL_SERVER_ERROR;

I admit, this is a blind alteration with possible adverse repercussions, so I don't vouch for it. Executing make once again yielded a new mod_ssl.so and this time httpd started just fine, this time with a strong key exchange added in. Testing the site with SSL Labs gave additional confirmation that SSL encryption was indeed working fine.

If wondering, I use Let's Encrypt for free SSL certificates. The recommended utility for obtaining certificates is certbot but that tool with its overly complex and finicky python virtual environment wouldn't work under FC14. The tool that worked beautifully was getssl. It's a simple and clean, yet powerful and flexible script written as one executable file in bash script. Kudos to the getssl team for creating this robust tool.

So there you have it for enabling modern SSL/TLS in an older environment, in this case FC14. The prevailing wisdom is to abandon the old OS and start off fresh with a newer platform. I don't disagree with that philosophy and I set out to do just that when I started on this journey. In the end, my way was possibly more difficult and more prone to pitfalls but ultimately it ended up being more satisfying and more instructive.

I haven't migrated the entire site to HTTPS yet, but you can click secure whoami to view and examine the first secure page.

 

What The Linux Ghost Bug Teaches

by @ 6:07 pm
Filed under: computers,hacking — Tags:

A couple of weeks ago it was revealed that a known Linux bug, Ghost (short-ish for the gethostbyname() function in the older glibc library versions) is riskier than previously thought. So the internet became abuzz with warnings to those who might not have updated their Linux distros.

I have several versions of Fedora running on various machines and updating them was simply not an option. Unfortunately they are also too old and patches are no longer available. But here comes the beauty of Linux, the open source code model. Combine that with a virtual server like Hyper-V and you have all the tools you need to create the patch yourself.

This is what I did to create patches for one of my platforms:

  • Created a guest virtual machine on the virtual server.
  • Downloaded the needed version of Fedora from this archive.
  • Installed the OS on the guest machine.
  • Downloaded the appropriate source code version of glibc. rpmfind.net is a good place to find many source code packages.
  • After installing all tools and libraries necessary to compile and build glibc, I used this StackExchange post as a guide to patch the C files based on the documented modifications and built the rpm package.
  • After installing and testing the newly built glibc library on the guest machine, I copied the rpm files to the production machine and installed them.
  • After a reboot, the bug was patched.

C code

Now many would object to running an older and unsupported version of Linux for production but I am not so sure that jumping to every new version as soon it is released contributes to additional safety. Staying with older versions does make the job of patching these sorts of bugs more cumbersome, but there's something to be said about the educational value of patching these bugs at more basic levels than just running the yum or apt-get commands. I, for one, learned quite a bit from this exercise.

 

Linux Shellshock Bash Bug Workaround

by @ 12:55 pm
Filed under: computers,hacking,internet — Tags: , ,

The warnings about the shellshock bash bug are ominous and not unfounded. This is perhaps a greater risk than Heartbleed. Here are the gory details of this bug.

To test your system for this bug run the following command from the shell:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

if you see the word 'vulnerable' anywhere in the output, like below, you have the bug.

shellshock bash bug

Because bash is such a fundamental part of Linux/Unix and used in so many ways and so prevalent, it wouldn't be that difficult for malicious hackers to use this bug to penetrate a machine and do all kinds of bad things including completely take over the machine. Web sites would the most obvious target of such attacks.

Now how to fix this. New bash versions with the bug patched have become available so users can update bash and be done. But this is not as easy to do for everyone. Some people may have older, obsolete versions of Linux, so they may not find the new patched bash version. They would need to get the source code and the patches, and then build and install it themselves. Yes, I know everyone should be on the latest version of everything, and I am guilty as charged, but let's dispense with the tarring and feathering for now.

Redhat however, in its haste and panic, had released a workaround on this page with a small block of C code that once installed, would disable function definitions and therefore mitigate this risk. They called it dangerous because one must assume this workaround would disable a legitimate feature of bash and possibly cause system failure if it were being used. Unfortunately a while later this workaround vanished (Update: actually here is the Redhat page for LD_PRELOAD mitigation. I don't know,  maybe the page never vanished at all. Just use the steps on that page then), but not before I had availed myself to it. For me, the ease and speed of its deployment made it worthy of a try. And here are the steps.

1- Put the following C code in a new file, bash_ld_preload.c.

#include <sys/types.h>
#include <stdlib.h>
#include <string.h>

static void __attribute__ ((constructor)) strip_env(void);
extern char **environ;

static void strip_env() {
  char *p,*c;
  int i = 0;

  for (p = environ[i]; p!=NULL;i++ ) {
    c = strstr(p,"=() {");
    if (c != NULL) {
      *(c+2) = '\0';
    }
    p = environ[i];
  }
}

2- Compile bash_ld_preload.c to get bash_ld_preload.so using the following command.

$ gcc bash_ld_preload.c -fPIC -shared -Wl,-soname,bash_ld_preload.so.1 -o bash_ld_preload.so

3- copy bash_ld_preload.so to the /lib/ directory like so:

$ cp bash_ld_preload.so /lib/

4- Add the following to the file /etc/ld.so.preload on a line by itself:

/lib/bash_ld_preload.so

5- Restart all relevant services or just reboot the system to be sure.

 

There you have it. I deployed this on several machines that run various applications. It killed the bug and there were no adverse effects. That means that those machines were not using the function definition feature of bash. Of course at some point we may write code or install applications that need to use this feature and if we have forgotten about this workaround, there will be a lot of head-scratching.

So, use the above workaround at your own risk. It will probably work for you, but the best approach as always is to update your platform and of course your version of bash.

Year 2038 problem

by @ 9:11 pm
Filed under: computers — Tags: , ,

25 years from now we could be dealing with an issue similar to the Y2K issue, year 2038 problem.

This problem was brought to my attention by  user 'Ken' commenting on the countdown tool page on this site. Basically *nix systems keep time in 32-bit integer formats counting seconds since Jan. 1, 1970. On Jan. 19, 2038 the 32-bit integer will overflow, resetting to 0 and many systems may interpret that as year 1901.

Certainly a vexing issue, but one with some time remaining to resolve. Even better, some of us will either be retired or simply no longer around to worry about it at all.

A number of fixes and workarounds have been proposed, chiefly among them using a 64-bit integer to keep time. That will do quite nicely and we won't have to worry about the rollover issue for some 292 billions years 🙂

Disabling SELinux

by @ 6:16 pm
Filed under: computers,hacking — Tags: ,

I know it's sacrilegious for some to disable a security feature on a platform, but SELinux (an enhanced Linux security feature) has left me no choice but doing exactly that on Linux.

SELinux was added to Linux to give it additional security measures beyond what it inherited From Unix. By default many of the Linux distros such as Fedora have SELinux built into their kernels and enabled upon install.

The issue is that SELinux can be so restricting and obsessive about curbing malicious activity that it can also hinder normal operations leading to server stress or errors. Having been bitten by SELinux multiple times, I have vowed to deactivate it every time I install Linux on a host. The one time I forgot to disable it, the Varnish server I have setup for my company nearly died taking the company's web site along for the ride. Looking inside the messages file, this arcane message is what I saw in prodigious numbers:

setroubleshoot: SELinux is preventing irqbalance from mmap_zero access on the memprotect Unknown. For complete SELinux messages. run sealert -l efce…

I know the security sticklers would accuse me of not setting up SELinux correctly and for the record SELinux is very configurable. But my most favorite setting for SELinux is disabling it in the /etc/selinux/config file by setting SELINUX=disabled.

I don't have the time nor the inclination to learn SELinux's every minutia, which may or may not protect my hosts completely anyways. The old fashion file permissions, file ownership, suexec, sudo, suid, running daemons with least privilege, and a good dose of firewalling is good enough for me. Feel free to disagree.

Fedora 16 Pain and Confusion

by @ 10:56 am
Filed under: computers — Tags: ,

When I recently installed Fedora 16 on an experimental server at work, it felt like dealing with a whole new platform. Prior to version 16, my last install was version 14 and things seem to have been in their familiar places. With 16 I was suddenly dumbfounded, like starting to learn Linux anew. I am not even referring to the GUI here, not a big fan. Windows is fine with me there.

For starters, there's the new Grub2 boot loader to learn. Then there's NetworkManager which tries to sniff out and configure everything when I was just happy with the plain old network startup. Related to that is what’s known as Consistent Network Device Naming. My ethernet adapter was suddenly labeled p1p7 instead of the familiar eth-0.

But the worst offender of all is the startup hell known as systemd. In a not too distant past the daemons and other startup processes were housed under /etc/init.d/ and then configured via chkconfig. Those were the good days of SysV. No more, now we have the purported super-polished, parallelized, speedy and advanced systemd, managed via the systemctl command. random symbolic linking here and there and strange file extensions. Even worse, old commands like chkconfig are hacked to call the newfangled systemd functions. I'm still trying to figure out the systemd craziness.

I know some of these changes started with previous versions and these changes were supposedly introduced to take advantage of new architectures and simplify, streamline, and accelerate operations, but that doesn't alleviate the shock and confusion of finding oneself in an unfamiliar terrain so unexpectedly.

Maybe I'm too old and set in my ways. And perhaps I should understand that Fedora is free and experimental and radical changes are par for the course. But there's a lesson Linux can learn from its nemesis, Windows. Microsoft hasn't scrapped Service Control Manager or Device Manager with every successive release of Windows. They improved them and added more features but the core functions remained the same and that goes a long way to allay users' and admins' fears of upgrading to new versions.

Powered by


Read Financial Markets  |   Home  |   Blog  |   Web Tools  |   News  |   Articles  |   FAQ  |   About  |   Privacy  |   Contact
Donate Bitcoin: 1K9TzBvQ2oaEb4tX9t2vKDtZouMcpfV6QF
paypal.me/rhashemian
© 2001-2018 Robert Hashemian   Powered by Hashemian.com