Zombies, ghosts and goblins will be walking the streets this Saturday as we celebrate Halloween, but the scariest creatures aren’t knocking on your door for trick or treat – they’re in your data center. Among the ghoulish guests:
- Zombie servers are plugged in and running, but aren’t doing any actual work. So they’re sucking up electricity, adding to cooling costs, and wasting scarce IT dollars.
- Ghost servers are machines that network administrators believe they are managing, but don’t actually exist. These apparitions can lead you down paths of inefficient capacity management and scare you into purchasing unneeded resources.
- Goblin servers are functioning, but have been set up without the networking team’s knowledge. Hidden from your view, these servers create compliance, reliability and security risks.
The three kinds of servers haunting IT, in other words, are the undead, the unreal and the unknown.
Sadly, this is not a Halloween joke. A recent story in The Wall Street Journal estimated there are 10 million zombie servers worldwide consuming four gigawatts of electricity – enough to power all the homes in New York City. Many of the companies afflicted with zombies don’t realize the server room is responsible for 30 to 50 percent of their entire electric bill. A large multi-national media company recently removed almost 15,000 zombie servers from its data centers and saved about $10 million annually in utility, maintenance, and licensing costs.
Infoblox customers confirm that ghost and goblin servers are also very real problems, and the situation is only getting worse as networks become bigger and more complex.
I don’t claim to be Buffy the Vampire Slayer, but I do have some tips for hunting zombie, ghost and goblin servers using the supernatural powers of Infoblox DDI.
Infoblox DNS, DHCP, and IP address management (DDI) appliances, enhanced with Infoblox Network Insight or Infoblox NetMRI, can reconcile the differences between your network records and what’s really being used on the network. The solution provides:
- Reporting on DNS query activity to identify servers with little or no network communication.
- Discovery of nodes and topology for reconciliation with records, identifying location, and troubleshooting.
- Server network activity monitoring.
- Network services discovery.
- Fingerprinting of all network devices and endpoints for proper classification and prioritization.
- Centralized view of allocated networks and reserved, used, and available IP addresses.
- Open, RESTful APIs for automated access to network and endpoint object database.
One way to find zombie servers is by monitoring Domain Name System (DNS) traffic. Little or no DNS query activity to or from a machine could indicate a zombie that is ready for removal. Infoblox DDI includes several out-of-the-box reports on DNS rates and trends across different time periods, along with DNS query response logs.
A sample Infoblox query response log.
Infoblox IP Address Management (IPAM) provides a centralized, global view of your IP address space. Ensuring that it matches the true state of the network is vital. This is where Infoblox Network Insight can help. Network Insight discovers network devices, endpoints and their connectivity, and continuously synchronizes with Infoblox IPAM. Discrepancies between the network authority and the IPAM authority are a warning sign of ghost or goblin servers. Examples of these discrepancies are:
- An unmanaged IP Address is discovered.
- A network is discovered, but it is not a managed network in IPAM.
- An IP address is marked as managed in IPAM, but it is not appearing on the network or its last discovery date is very old.
- A host IP Address is marked as being connected to a different switch port than the one it is discovered on.
One key indicator of a zombie server is little or no network activity. Infoblox NetMRI tracks performance metrics such as transmit and receive counts, rates, and percentage. For instance, you could pull a list of servers that have had an inbound byte count below a specified threshold to be evaluated as possible zombies. The servers may still be pingable and reconcilable as a managed hosts in your IPAM authority, but they’re not providing any value, they’re causing the utility bill to go up, they’re taking space and they’re wasting switch port capacity.
Infoblox NetMRI also discovers TCP/IP services in use. A particularly useful service to monitor is bootps (port 67), which could indicate an unknown DHCP server on the network. Reconciling these discovered DHCP services with the DHCP servers defined in Infoblox DDI not only ensures you don’t have a goblin server on the network, but that you don’t have a rogue DHCP server giving out conflicting and/or unmanaged IP Addresses.
Finally, when hunting zombies, ghosts and goblins, it’s important to have a classification of endpoint types. You may have thousands of endpoints on your network, and if you identify hundreds or even thousands as potential haunted devices, you need a way to figure out which are actually servers and which of these servers should be considered for exorcism. Infoblox has three solutions to identify endpoints:
- Network Insight, which discovers network devices, endpoints and the connectivity between them.
- DHCP fingerprinting, which uses the DHCP process to determine an endpoint device type.
- An IPAM database of data center and network objects. Metadata tagging (extended attributes) on the IPAM objects can make it easier to find the location, purpose or owner of servers.
One important disclaimer: Some zombie servers have no network connection, and therefore can’t be found by any type of network monitoring software. Instead, IT administrators need to unearth these disconnected demons by physically walking through the aisles of their data centers.
Indeed, there is no silver bullet for finding zombie, ghost and goblin servers. Yet that doesn’t mean you have to rely on magic spells. We at Infoblox believe one of the best ways to find servers hiding in dark corners is by shining the bright light of network discovery.
Our best wishes for a happy Halloween, free from frightful ghouls in your IT infrastructure.
