Sometimes one could become so comfortable with something, that one becomes blind to the underlying technology. Such was the case with me a few days ago when I was trying to delete an IP address from one Linux node on the network and assign it to another.
The problem was that the new node would remain inaccessible on the network. It would eventually show up, but that didn't make troubleshooting easy. We're all so used to plugging a node into a switch and have it up and running that we forget that underneath the IP address, there's the MAC address with the DNS-like ARP tables running on switches and nodes.
Apparently the "service network reload" command on the Linux box wasn't making any ARP announcements on the network, leaving the ARP tables (evidently with long aging timers) with old mappings. And that explains why the new Linux node would remain inaccessible for some time.
I'm not sure if the network subsystem in Linux is supposed to advertise a new IP to MAC (possibly using a an ARP request). Strangely, even a reboot wouldn't fix this problem. It is possible that the firewall (iptables) rules were preventing this. Whatever the case, a manual ARP request using the arping command seemed to have resolved this. Here's the syntax (with a phony IP):
arping -U -I eth0 192.168.100.100
arping is a useful Linux tool similar to the ping command, but operating at the MAC level. I suppose there's a good chance that even pinging a node on the local network from the Linux box would have done the trick and updated the ARP tables. Anyways, if you find yourself in a similar situation, check the ARP tables on your switches. They're so easy to forget.