|
|
December 8, 2009
From a development standpoint, cloud computing is a flexible and elastic computing environment. Sort of a server with unbound resources capable of running infinite programs for infinite users, with infinite processing power, memory, storage, and bandwidth.
Cloud computing is a dream for people like me who are developers at heart but spent much of their time as involuntary administrators, be it on Web servers, database servers, file servers, firewalls, networks, or whatever.
Of course everything good must come with something bad. Cloud computing is no exception. There is the lack of complete control over the resources, the risk of unforeseen outages and the risk of data leaks and data loss. And being at the mercy of the cloud vendors may not be good either. They could just shut you down by mistake or because of some bogus reason.
But one aspect I hadn't given much thought about is cost, specially the unexpected ones. Last weekend at work an unforeseen condition awoke a piece of bad code in our data access layer causing CPU thrashing on our database server. The systems still worked, albeit near a denial-of-service condition. By Monday we identified and patched the code and everything was back to normal. No cost was incurred as a result, other than perhaps some extra heat generated by the server.
If our systems were hosted on a cloud we might have been faced with a large invoice from the provider for the processing and bandwidth used. Or the provider might have just shut down our services to protect their other clients. The results could have been catastrophic.
When you operate your own servers, you may be able to get away with some mistakes. But in the cloud mistakes could cost you a lot, even your job. That risk is something we should be keenly aware of, as the computing landscape slowly shifts to the cloud. Like it or not, cloud computing will eventually usurp customized systems. It'll come with many benefits, but we must recognize that it won't be as easy as dumping in the applications and calling it a day.
cloud computing,servers
November 13, 2009
Bad reputation. Lawyers have it, used-car salesmen have it, priests and clerics have it. And now computer programmers? That's what I thought when I read about the two programmers working for Bernard Madoff's Ponzi investment firm being arrested and charged with fraud.
It seems that their skills were put to good use by fabricating reports and statements for clients and the SEC, making them look legitimate. According to the story, they both knew that they were engaged in shady work, and they both had complained about it. But in return for a mediocre bonus from the company they'd stopped complaining and had silently continued manipulating the numbers and generating phantom reports.
As a programmer myself this story gave me a pause. What would I have done? I'm not the blackmailing type, but would I have just drawn a salary and turned a blind eye? Would I have raised my concerns, not to extort, but out of moral obligation or at least out of concern for my own complicity and the possible punishment? Or would I have left the company without leaking a word, just to save my own skin?
Businesses ask many things of their employees. Some are blatant violations of ethics, but some others fall into gray areas. I wonder if these two programmers could have avoided the hot water if they had kept their mouths shut and just did what they were asked. The fact that they raised concerns only to bury them after payoffs has certainly been a factor in what they're facing now.
At any rate, I have a good feeling that if Madoff's scheme hadn't turned into a high profile case that it is today, these programmers would have never been implicated. There's just too much public fury and too few targets to go after. I mean how many programmers at Enron have suffered the same fate? As far as I know, none.
madoff,enron, securities fraud,wall street,programmers
Bad reputation. Lawyers have it, used-car salesmen have it, priests and clerics have it. And now computer programmers? That's what I thought when I read about the two programmers working for Bernard Madoff's Ponzi investment firm being arrested and charged with fraud.
It seems that their skills were put to good use by fabricating reports and statements for clients and the SEC, making them look legitimate. According to the story, they both knew that they were engaged in shady work, and they both had complained about it. But in return for a mediocre bonus from the company they'd stopped complaining and had silently continued manipulating the numbers and generating phantom reports.
As a programmer myself this story gave me a pause. What would I have done? I'm not the blackmailing type, but would I have just drawn a salary and turned a blind eye? Would I have raised my concerns, not to extort, but out of moral obligation or at least out of concern for my own complicity and the possible punishment? Or would I have left the company without leaking a word, just to save my own skin?
Businesses ask many things of their employees. Some are blatant violations of ethics, but some others fall into gray areas. I wonder if these two programmers could have avoided the hot water if they had kept their mouths shut and just did what they were asked. The fact that they raised concerns only to bury them after payoffs has certainly been a factor in what they're facing now.
At any rate, I have a good feeling that if Madoff's scheme hadn't turned into a high profile case that it is today, these programmers would have never been implicated. There's just too much public fury and too few targets to go after. I mean how many programmers at Enron have suffered the same fate? As far as I know, none.
madoff,enron, securities fraud,wall street,programmers
June 5, 2009
A few years ago when I needed to synchronize large collection of files for a number of Windows Server 2003 hosts I stumbled upon a Windows Resource Kit utility named Robocopy. This is a great file synchronization tool with lots of switches that can quickly copy entire folders from one Windows host to another along with their NTFS data such as dates and security details, yielding exact duplicates.
There are a number of tools on the market that can do the job, but Robocopy is free, fast, and easy to use. I had a number of hosts auto-synchronized using Robocopy in scheduled tasks and they worked admirably without any hassles. It's one of those set-it-and-forget-it utilities that once configured, it doesn't need any further attention.
Apparently Robocopy was such a useful tool that Microsoft decided to bundle it with Windows Vista and Windows Server 2008. There's even a GUI interface for it for those less inclined to run it from a command prompt. The newer bundled version may have a few extra switches but is otherwise almost identical to the previous versions that were part of the Resource Kit. There's also one more difference that is perhaps less noticeable. It runs substantially slower that the past versions.
I discovered the performance issue today while doing a routine check on the scheduled tasks I had configured for our newer Windows Server 2008 hosts. It appeared that the Robocopy tasks I had set up on the new servers were taking considerably longer time to finish than comparable tasks configured on Windows Server 2003 hosts. The copy operations were still being performed flawlessly but the completion times were drastically longer, specially for servers with large numbers of folders and files.
I spent a number of hours investigating this issue on multiple servers. I took various measurements, tweaked networking parameters, and used various Robocopy switches and values, but no matter what I did the performance issue kept persisting.
The Solution
Out of options and with no solutions at hand, I made one final attempt that I hadn't thought of before. I copied an older version of Robocopy (a version that came with the Windows Server 2003 Resource Kit) to one of the Windows Server 2008 hosts and proceeded to synchronize folders using that version. To my amazement, the copy operation completed as fast as it used to do under Windows Server 2003. We're not talking a small improvement here. The job ran over 40 times faster than it had using the newer bundled copy of Robocopy. Incredulous, I tested this multiple times on multiple servers and the results were the same. The older version of Robocopy dramatically outperformed the newer version even when executed on Windows Server 2008.
There you have it. If you are not happy with the speed of Robocopy on Windows Vista or Windows Server 2008, give the older version, included with Windows Server 2003 Resource Kit, a try and see if it does the same for you as it did for me.
There are a couple of caveats here you should be aware of. First, I checked and rechecked my results quite extensively but I wouldn't claim that to be exhaustive. Do your own detailed verification before you let the old Robocopy loose on your hosts. Second, I ran the old version of Robocopy (a 32-bit image) on 64-bit Windows Server 2008 hosts and as mentioned, I had great results. But that version of Robocopy is not even supported on 64-bit Windows Server 2003, let alone Windows Server 2008. So I suppose there is a slight inherent risk of malfunction. For me, that risk is worth the performance gain, but your risk tolerance may be lower.
robocopy,windows,windows server,microsoft,windows resource kit,vista
November 13, 2008
If you use the Response.TransmitFile method in your ASP.NET pages, there is a good chance that the file being transmitted is locked and unavailable for writing during that operation. TransmitFile was introduced with .NET V2.0 and it's supposed to be a more optimized version of WriteFile.
For months I had been dogged with exception alerts when a scheduled program was trying to update files used in TransmitFile. As the alerts were intermittent and the files would eventually get updated by the program, I would just ignore the messages. But today I decided to take a closer look and discovered that TransmitFile is to blame for the files being locked.
As I understand, TransmitFile in ASP.NET makes use of the Windows TransmitFile API function which directly streams a file to an open socket. It is more optimized than WriteFile which buffers the entire content of the file before emitting it. In that sense, WriteFile consumes precious memory and processing resources, switching between kernel and user modes, to accomplish its work.
While TransmitFile API function is optimized, care must be taken to avoid having the streamed data getting mixed with other data and my hunch is that some locking mechanism is used to assure an orderly and sequential transmission of data under ASP.NET. That's all fine until one needs to write to the file which could get shot down while the lock is still active. I was able to reproduce this behavior in a loop where a file is continuously updated and then streamed via TransmitFile. Here’s a simple example:
void Page_Load() { Response.Buffer=false; for(;;) { System.IO.File.WriteAllText(MapPath("/test.txt"), DateTime.Now.ToString()); System.Threading.Thread.Sleep(100); Response.TransmitFile(MapPath("/test.txt")); Response.Write("<br>"); }}
Running this ASP.NET page invariably lead to an exception being thrown. I then changed the code as shown below:
void Page_Load() { Response.Buffer=false; for(;;) { System.IO.File.WriteAllText(MapPath("/test.txt"), DateTime.Now.ToString()); System.Threading.Thread.Sleep(100); // CHANGED LINE BELOW Response.Write(System.IO.File.ReadAllText(MapPath("/test.txt"))); Response.Write("<br>"); }}
I ran the page a number of times and no exception was thrown. I'm sure that ReadAllText is nowhere as optimized as TransmitFile, but I would readily take the performance penalty over the unpredictable exceptions. Incidentally, WriteFile proved to be no better than TransmitFile in locking the file.
Based on the simple test I have concluded that TransmitFile or WriteFile should be used when the file is never or rarely updated. If the file is to be updated frequently, ReadAllText (or other System.IO file reading operations) is the way to go. But if someone can disprove my tenuous theory, by all means.
asp.net,file systems,windows,web programming
August 28, 2008
While programming is my main focus at my company, one of my side jobs at work is networking. I have no complaints as I'm curious and interested in the inner workings of computer networks. Our IT department handles most of the networking tasks, but I usually find myself getting involved in setting up connectivity in the company. Whether it's a firewall, a router, a reverse proxy, or a DNS server, I find the networking field too fascinating to ignore.
That's why when the latest DNS vulnerability, discovered by Dan Kaminsky, came to light in April 2008, I began investigating our DNS servers to determine the risk factors. Dan's site contains a simple tool to assess the risk and it identified all of our caching DNS servers as vulnerable. A patch from Microsoft took care of our Windows-based DNS servers, but there was also a Fedora server in the mix running an old version of BIND that needed attention. Patching that server would have required upgrading to a newer version of Fedora.
I knew I could buy some time using the safety-in-numbers logic, but today I finally decided to tackle that server and plug the hole. My intention was to install the newest version of Fedora (version 9) on a new hardware and then add a patched version of BIND on top. BIND is a great name server product but it has a large footprint that seems like an overkill as a caching server. There are several other free DNS products out there so I began to look for an alternative.
My search eventually led me to PowerDNS (PDNS) and I decided to give that product a try. After installing Fedora 9 on the server, I downloaded the latest RPM of PNDS and promptly installed it on the server. PDNS comes in two flavors. The authoritative version and the caching version, known as Recursor which is the one I was interested in. The install was a breeze and the configuration was as easy as importing some of the data from the old BIND server and making some quick edits to the recursor.conf file. A server restart to make sure everything is in order, and I had the new caching server up and running, resolving names.
PDNS has been free of the DNS cache poisoning vulnerability for a few years now, and Dan's site confirmed that the new server was indeed running at much safer levels.
There is little doubt that the bad guys are hard at work to poison as many DNS servers as they can get their hands on. If your unpatched servers haven't been targeted yet, it's only a matter of time before they are. Whatever method or product you use to avert this risk, the sooner you do it, the better. As a quick alternative, you can use one of several free and already safe services like the one offered by OpenDNS.com as direct name servers or as forwarders on your caching servers.
dns,dns vulnerability,dns hack,fedora,bind,powerdns,opendns,networks,named
June 29, 2008
 Among some of the new classes introduced in Version 3.5 of the .NET Framework Class Library (FCL) were the syndication-related classes. While other technologies such as LINQ, extension methods and lambda expressions have been grabbing most of the attention the new syndication classes also deserve a nod. Part of the System.ServiceModel.Syndication namespace the classes offer a variety of methods to easily generate or consume syndication feeds in RSS 2.0 or ATOM 1.0 formats.
It's not that reading or writing feeds were exceedingly difficult before. FCL comes with a number of XML classes that facilitate working with XML data which all syndication feeds emanate from. No doubt there are plenty of sample code out there that made the task as easy as copy, paste and tweak. But now FCL comes with its own native classes to handle feeds, with advanced settings, intellisense, and potential of extension.
To demonstrate ease of use, here's a sample code that pulls in a sample feed from Google, and scrapes and saves the content of each link to a file:
var wc = new WebClient();using (var rss = XmlReader.Create( "http://finance.google.com/finance?morenews=10&q=NASDAQ:INTC&output=rss")) { var feed = SyndicationFeed.Load(rss); foreach (var x in feed.Items) { var uri = x.Links.Last().Uri; wc.DownloadFile(uri, @"c:\rss\" + Regex.Replace(uri.LocalPath, @"^.*/", "")); }}
That was easy, eh? Just remember to add System.ServiceModel.Web.dll as a reference to your project. Happy syndicating.
rss,atom,microsoft,net,asp.net,xml
June 18, 2008
Today I was trying to reach 1&1's home page, but the browser kept failing to pull up the site. Mysteriously I was able to reach 1&1's home page when I changed my DNS servers to those of OpenDNS.org. Feeling curious I decided to investigate the matter in depth. My default DNS server was reporting the IP address of www.1and1.com to be 217.160.232.1. While that address belongs to 1&1, it's really one of their routers or gateways and not a Web server. No wonder I was unable to access the site. the working IP address reported by OpenDNS.org and a number of other DNS servers was 217.160.226.203. That is indeed the correct IP address for www.1and1.com. So why was I seeing different results from different DNS servers?
As you may know the job of translating a host name to an IP address falls on a program known as the resolver which queries its designated DNS server for the answer. If the DNS server can not produce the translation (from its cache or authority zone), it issues what it's know as a recursive query to the DNS network on the Internet. The host name is broken to its fragments and each fragment from right to left is queried successively. The results generally consists of hosts known as NameServers, which get the query one step closer to the final result. The final NameServers produce the IP address translation. However, if any of the NameServers along the way can produce the translation, the query stops and the IP address is sent back to the resolver.
Using the Unix/Linux dig command I followed the name resolution for www.1and1.com one step at a time. Results are shown here and shortened for brevity.
This command displays the root servers:
# dig;; ANSWER SECTION:. 451081 IN NS M.ROOT-SERVERS.NET.. 451081 IN NS A.ROOT-SERVERS.NET.. 451081 IN NS B.ROOT-SERVERS.NET.. 451081 IN NS C.ROOT-SERVERS.NET.
This command queries one of the root servers and produces NameServers for "com." TLD (Top Level Domain):
# dig +norec @A.ROOT-SERVERS.NET www.1and1.com;; AUTHORITY SECTION:com. 172800 IN NS K.GTLD-SERVERS.NET.com. 172800 IN NS L.GTLD-SERVERS.NET.com. 172800 IN NS M.GTLD-SERVERS.NET.com. 172800 IN NS A.GTLD-SERVERS.NET.
This command queries one of the "com." NameServers:
# dig +norec @A.GTLD-SERVERS.NET www.1and1.com;; ANSWER SECTION:www.1and1.com. 172800 IN A 217.160.232.1;; AUTHORITY SECTION:1and1.com. 172800 IN NS ns27.1and1.com.1and1.com. 172800 IN NS ns28.1and1.com.
Generally the previous command shouldn't produce and IP address, instead the authority section would prompt a final query to one of the 1and1.com NameServers (which by the way have the correct IP translation.) Instead somehow an IP address is produced at this level and the query ends with this inaccurate IP translation. I've tried the same query with the homepage URL's of Microsoft, Google, Yahoo and a few other sites and none return an IP address at this level.
It remains to be seen if this erroneous translation would eventually spread around, causing 1&1's homepage to become widely inaccessible. Anyone knows how that IP translation ended up in of the "com." NameServers? Am I making wrong assumptions here? Feel free to let me know.
dns,nameservers,domain names,ip addresses,internet
January 4, 2008
Happy new year. Not so much for me. The new year was marked with a wicked lower back pain. An irritating bulging disk that gets inflamed every now and then and pinches the nerve, radiating pain everywhere. Guess that's just nature's way of manifesting my age to me. Yes I know, I'm not in my twenties anymore. Haven't been there for quite some time and the longer I live the further away I get.
So what do? Just add the back pain to the foot pain, added to the hamstring pain, added to the knee pain, and I have a nice variety of aches and pains.
I suppose pain is body's exception system. If you're a programmer, you know what I'm talking about. An exception in a program is raised when something totally unexpected happens in the running code and needs immediate attention. Good coding practice dictates that programmers anticipate and compensate for all possible errors before their code is hit by an exception. But sometimes there's no avoiding it.
- A number gets divided by zero - an exception is raised.
- A missing file is referenced - an exception is raised.
- A piece of data doesn't fit inside a database table column - an exception is raised.
- An XML stream is missing a tag - an exception is raised.
Graceful code is supposed to catch the exception, alert the user, and halt. After all something catastrophic must have happened and continuing the program could mean entering an invalid and unknown state. I admit, I've broken that rule a few times by catching an exception, logging the issue, and continuing as if nothing had happened. Why should proper programs get all the running privileges?
Body pain works the same way. It's a signal that something's gone wrong and needs attention. One must correct the problem before continuing with normal activities. That's exactly what I had intended to do. Give my back a few days of rest before getting back to running again. Except that last night I saw a jogger in the freezing temperature and my jealousy meter went off the scale.
So tonight I resolved to go for a walk, only to naturally speed up to a jogging pace after a few steps. And thus I entered into the invalid state of pain, as in, not knowing how my back will feel tomorrow. Oh well, why should only healthy, pain-free people have the privilege of running?
jogging,running,exceptions,programming,exception handling
December 24, 2007
Sometimes one could become so comfortable with something, that one becomes blind to the underlying technology. Such was the case with me a few days ago when I was trying to delete an IP address from one Linux node on the network and assign it to another.
The problem was that the new node would remain inaccessible on the network. It would eventually show up, but that didn't make troubleshooting easy. We're all so used to plugging a node into a switch and have it up and running that we forget that underneath the IP address, there's the MAC address with the DNS-like ARP tables running on switches and nodes.
Apparently the "service network reload" command on the Linux box wasn't making any ARP announcements on the network, leaving the ARP tables (evidently with long aging timers) with old mappings. And that explains why the new Linux node would remain inaccessible for some time.
I'm not sure if the network subsystem in Linux is supposed to advertise a new IP to MAC (possibly using a an ARP request). Strangely, even a reboot wouldn't fix this problem. It is possible that the firewall (iptables) rules were preventing this. Whatever the case, a manual ARP request using the arping command seemed to have resolved this. Here's the syntax (with a phony IP):
arping -U -I eth0 192.168.100.100
arping is a useful Linux tool similar to the ping command, but operating at the MAC level. I suppose there's a good chance that even pinging a node on the local network from the Linux box would have done the trick and updated the ARP tables. Anyways, if you find yourself in a similar situation, check the ARP tables on your switches. They're so easy to forget.
linux,arp,mac address,ethernet,ip address,network switch
« Newer Posts — Older Posts »
Powered by 
Read Financial Markets |
Home |
Blog |
Web Tools |
News |
Articles |
FAQ |
About |
Contact
© 2001-2012 Robert Hashemian
|
|
Liked this page?
Please consider creating a link to it
from your Web site.
|
|
|
r!ncRbBPwbNwsjS!~~_35.JPG)
NBPv5)ozWrw~~_35.JPG)
