Hashemian Blog
Web Tools, Financial Markets, Technology
Thursday, August 28, 2008
DNS Vulnerability
While programming is my main focus at my company, one of my side jobs at work is networking. I have no complaints as I'm curious and interested in the inner workings of computer networks. Our IT department handles most of the networking tasks, but I usually find myself getting involved in setting up connectivity in the company. Whether it's a firewall, a router, a reverse proxy, or a DNS server, I find the networking field too fascinating to ignore.
That's why when the latest DNS vulnerability, discovered by Dan Kaminsky, came to light in April 2008, I began investigating our DNS servers to determine the risk factors. Dan's site contains a simple tool to assess the risk and it identified all of our caching DNS servers as vulnerable. A patch from Microsoft took care of our Windows-based DNS servers, but there was also a Fedora server in the mix running an old version of BIND that needed attention. Patching that server would have required upgrading to a newer version of Fedora.
I knew I could buy some time using the safety-in-numbers logic, but today I finally decided to tackle that server and plug the hole. My intention was to install the newest version of Fedora (version 9) on a new hardware and then add a patched version of BIND on top. BIND is a great name server product but it has a large footprint that seems like an overkill as a caching server. There are several other free DNS products out there so I began to look for an alternative.
My search eventually led me to PowerDNS (PDNS) and I decided to give that product a try. After installing Fedora 9 on the server, I downloaded the latest RPM of PNDS and promptly installed it on the server. PDNS comes in two flavors. The authoritative version and the caching version, known as Recursor which is the one I was interested in. The install was a breeze and the configuration was as easy as importing some of the data from the old BIND server and making some quick edits to the recursor.conf file. A server restart to make sure everything is in order, and I had the new caching server up and running, resolving names.
PDNS has been free of the DNS cache poisoning vulnerability for a few years now, and Dan's site confirmed that the new server was indeed running at much safer levels.
There is little doubt that the bad guys are hard at work to poison as many DNS servers as they can get their hands on. If your unpatched servers haven't been targeted yet, it's only a matter of time before they are. Whatever method or product you use to avert this risk, the sooner you do it, the better. As a quick alternative, you can use one of several free and already safe services like the one offered by OpenDNS.com as direct name servers or as forwarders on your caching servers.
dns,dns vulnerability,dns hack,fedora,bind,powerdns,opendns,networks,namedLabels: computers, hackers, networks < DNS Vulnerability>
// posted by rh
Wednesday, June 18, 2008
DNS Mystery, NameServers, IP addresses
Today I was trying to reach 1&1's home page, but the browser kept failing to pull up the site. Mysteriously I was able to reach 1&1's home page when I changed my DNS servers to those of OpenDNS.org. Feeling curious I decided to investigate the matter in depth. My default DNS server was reporting the IP address of www.1and1.com to be 217.160.232.1. While that address belongs to 1&1, it's really one of their routers or gateways and not a Web server. No wonder I was unable to access the site. the working IP address reported by OpenDNS.org and a number of other DNS servers was 217.160.226.203. That is indeed the correct IP address for www.1and1.com. So why was I seeing different results from different DNS servers?
As you may know the job of translating a host name to an IP address falls on a program known as the resolver which queries its designated DNS server for the answer. If the DNS server can not produce the translation (from its cache or authority zone), it issues what it's know as a recursive query to the DNS network on the Internet. The host name is broken to its fragments and each fragment from right to left is queried successively. The results generally consists of hosts known as NameServers, which get the query one step closer to the final result. The final NameServers produce the IP address translation. However, if any of the NameServers along the way can produce the translation, the query stops and the IP address is sent back to the resolver.
Using the Unix/Linux dig command I followed the name resolution for www.1and1.com one step at a time. Results are shown here and shortened for brevity.
This command displays the root servers:# dig ;; ANSWER SECTION: . 451081 IN NS M.ROOT-SERVERS.NET. . 451081 IN NS A.ROOT-SERVERS.NET. . 451081 IN NS B.ROOT-SERVERS.NET. . 451081 IN NS C.ROOT-SERVERS.NET. This command queries one of the root servers and produces NameServers for "com." TLD (Top Level Domain):# dig +norec @A.ROOT-SERVERS.NET www.1and1.com ;; AUTHORITY SECTION: com. 172800 IN NS K.GTLD-SERVERS.NET. com. 172800 IN NS L.GTLD-SERVERS.NET. com. 172800 IN NS M.GTLD-SERVERS.NET. com. 172800 IN NS A.GTLD-SERVERS.NET. This command queries one of the "com." NameServers:# dig +norec @A.GTLD-SERVERS.NET www.1and1.com ;; ANSWER SECTION: www.1and1.com. 172800 IN A 217.160.232.1 ;; AUTHORITY SECTION: 1and1.com. 172800 IN NS ns27.1and1.com. 1and1.com. 172800 IN NS ns28.1and1.com. Generally the previous command shouldn't produce and IP address, instead the authority section would prompt a final query to one of the 1and1.com NameServers (which by the way have the correct IP translation.) Instead somehow an IP address is produced at this level and the query ends with this inaccurate IP translation. I've tried the same query with the homepage URL's of Microsoft, Google, Yahoo and a few other sites and none return an IP address at this level.
It remains to be seen if this erroneous translation would eventually spread around, causing 1&1's homepage to become widely inaccessible. Anyone knows how that IP translation ended up in of the "com." NameServers? Am I making wrong assumptions here? Feel free to let me know.
dns,nameservers,domain names,ip addresses,internetLabels: internet, networks, web < DNS Mystery, NameServers, IP addresses>
// posted by rh
Monday, December 24, 2007
ARP, arping, and MAC
Sometimes one could become so comfortable with something, that one becomes blind to the underlying technology. Such was the case with me a few days ago when I was trying to delete an IP address from one Linux node on the network and assign it to another.
The problem was that the new node would remain inaccessible on the network. It would eventually show up, but that didn't make troubleshooting easy. We're all so used to plugging a node into a switch and have it up and running that we forget that underneath the IP address, there's the MAC address with the DNS-like ARP tables running on switches and nodes.
Apparently the "service network reload" command on the Linux box wasn't making any ARP announcements on the network, leaving the ARP tables (evidently with long aging timers) with old mappings. And that explains why the new Linux node would remain inaccessible for some time.
I'm not sure if the network subsystem in Linux is supposed to advertise a new IP to MAC (possibly using a an ARP request). Strangely, even a reboot wouldn't fix this problem. It is possible that the firewall (iptables) rules were preventing this. Whatever the case, a manual ARP request using the arping command seemed to have resolved this. Here's the syntax (with a phony IP):arping -U -I eth0 192.168.100.100 arping is a useful Linux tool similar to the ping command, but operating at the MAC level. I suppose there's a good chance that even pinging a node on the local network from the Linux box would have done the trick and updated the ARP tables. Anyways, if you find yourself in a similar situation, check the ARP tables on your switches. They're so easy to forget.
linux,arp,mac address,ethernet,ip address,network switchLabels: computers, networks < ARP, arping, and MAC>
// posted by rh
Tuesday, October 23, 2007
Proxy Server Woes
Troubleshooting misbehaving Web applications is a part of my job at work. Most of the time it's some buggy code that I or someone in our team had written. But sometimes I get approached about system problems too, like when a browser is having connection issues and can't access Web sites. If you are in the help desk business you know how the question is posed, "Is the Internet down?". I've always been tempted to reply: "Why yes, it is down. All that redundancy worked into it back in the DARPA era was a pile of lies. The whole Internet is toast."
Most of the time browser problems are not browser problems at all, they are network issues. The browser just gets the blame because it's front and center. A bad cable kicked once too many, an unplugged Wi-Fi access point, or a beleaguered DHCP server are the usual suspects. But the one that trips me most of the time is the proxy server setting. I've gotten better at it but I still occasionally miss this little item from my checklist.
As you may know proxy servers are protocol-level servers (appliances) that work as brokers between clients and servers. The most commonly used is the Web (HTTP) proxy server. Traffic passing back and forth between browsers and servers is funneled through them. Why use them? A detailed discussion is too boring, but in a nutshell, they make network admins' lives easier (more granular control, central management, etc), and they can cache and compress data saving on bandwidth and thus speeding things along.
Proxy servers are helpful, as long as one remembers that they're being used, especially when troubleshooting. Case in point, the other day I was working on a laptop connected to the Internet over Verizon Wireless broadband EVDO. No matter what I did I couldn't browse to the laptop's local Web server. Everything was fine when wireless was off, but as soon as I started it up, the local Web server would become inaccessible. What was happening? I could successfully ping my own server (duh), the Web service was running fine and listening on the local address, and there was no proxy server setting hidden in my browser (obvious place to look, but easily missed, specially when frantic).
After hours of investigating, including recruiting assistance from a colleague, the issue was revealed, quite by accident. Verizon Wireless comes with an add-on product called Venturi (enabled by default), used for optimization and compression. Turns out Venturi was the cause of this issue. When disabled, everything began working as expected. Venturi inserts itself in the TCP/IP stack and reroutes Web requests to a proxy server. No wonder I couldn't browse to the local server, the external Venturi proxy server had no idea how to access it, nor anything else on the private LAN.
The trouble with proxy servers is that it's hard to detect that your applications are using one, unless you pay attention to the telltale signs. First, with most applications, including browsers, you can check the network options and see if a proxy server address is configured. Next you can visit an IP address and header lookup page like whoami/My IP Address on this site, and check out the results. This page and the likes will display whatever information they can glean from your connection. Sometimes, but not always, proxy servers add headers such as "X-Forwarded-For" or "Via" to reveal the actual source of the request. You can also compare the shown source IP address to the IP address of your own machine. From the command line enter the "ipconfig" command (in Windows) which will display your actual IP address. If the lookup page is displaying the same IP address, chances are there is no proxy server in between, otherwise your traffic may be going through a proxy server and that would explain the irregularities of the sorts I was experiencing.
proxy servers,verizon wireless,evdo,ip address,ipconfig,tcp/ip,networks,browsers,httpLabels: networks, technology, web < Proxy Server Woes>
// posted by rh

|
Links
Technorati Profile
TMCnet.com
ARCHIVES
09/01/2003 - 10/01/200303/01/2004 - 04/01/200404/01/2004 - 05/01/200405/01/2004 - 06/01/200406/01/2004 - 07/01/200407/01/2004 - 08/01/200408/01/2004 - 09/01/200409/01/2004 - 10/01/200410/01/2004 - 11/01/200411/01/2004 - 12/01/200412/01/2004 - 01/01/200501/01/2005 - 02/01/200502/01/2005 - 03/01/200503/01/2005 - 04/01/200504/01/2005 - 05/01/200505/01/2005 - 06/01/200506/01/2005 - 07/01/200507/01/2005 - 08/01/200508/01/2005 - 09/01/200509/01/2005 - 10/01/200510/01/2005 - 11/01/200511/01/2005 - 12/01/200512/01/2005 - 01/01/200601/01/2006 - 02/01/200602/01/2006 - 03/01/200603/01/2006 - 04/01/200604/01/2006 - 05/01/200605/01/2006 - 06/01/200606/01/2006 - 07/01/200607/01/2006 - 08/01/200608/01/2006 - 09/01/200609/01/2006 - 10/01/200610/01/2006 - 11/01/200611/01/2006 - 12/01/200612/01/2006 - 01/01/200701/01/2007 - 02/01/200702/01/2007 - 03/01/200703/01/2007 - 04/01/200704/01/2007 - 05/01/200705/01/2007 - 06/01/200706/01/2007 - 07/01/200707/01/2007 - 08/01/200708/01/2007 - 09/01/200709/01/2007 - 10/01/200710/01/2007 - 11/01/200711/01/2007 - 12/01/200712/01/2007 - 01/01/200801/01/2008 - 02/01/200802/01/2008 - 03/01/200803/01/2008 - 04/01/200804/01/2008 - 05/01/200805/01/2008 - 06/01/200806/01/2008 - 07/01/200807/01/2008 - 08/01/200808/01/2008 - 09/01/200809/01/2008 - 10/01/200810/01/2008 - 11/01/200811/01/2008 - 12/01/200812/01/2008 - 01/01/200901/01/2009 - 02/01/200902/01/2009 - 03/01/200903/01/2009 - 04/01/200904/01/2009 - 05/01/200905/01/2009 - 06/01/200906/01/2009 - 07/01/200907/01/2009 - 08/01/200908/01/2009 - 09/01/200909/01/2009 - 10/01/200910/01/2009 - 11/01/200911/01/2009 - 12/01/2009
|