By Beth Cohen
“We are not only downsizing but we are also stupefying when we lay off the senior network engineers.”
So said an anonymous executive at a Fortune 100 company. Unfortunately, that means that you are now expected to handle sophisticated network analysis issues in addition to the daily grind of keeping the network running smoothly.
This has never happened to you. You know that your coworkers are using instant messages to their friends. They are using their private accounts, but your management has just asked you to put a stop to it. They allege it is costing the company productivity and bandwidth. How can you find out if their claims are true? In addition, what can you do about it now?
Your manager has told you to determine why the network connection to your branch office in Kalamazoo runs so slowly. Is there a network traffic jam or is something else causing the problem? Nobody else has complained about the issue. When the network has problems, do you have the tools to detect them so you can respond appropriately?
As the company security officer (in addition to all of your other hats), a law enforcement agency has approached you. Someone was attempting to crack the Pentagon computer systems using your company’s network. They would like you to help track down who it was. Can you do anything? Is it possible that your network was hijacked and you did not know it?
Welcome to the world of network and systems analysis tools. With a good network and systems analyzer, you can know exactly what traffic is on your network and precisely how your applications and systems are used. You can also use monitoring tools to solve network problems in real time – critical in today’s fast-paced 7×24 enterprises. The newer network analysis tools help you solve your network and systems headaches and can guide your business’s network and application growth but they are sophisticated tools that take some training to understand and use properly.
Monitoring or analysis – What is the difference?
You are responsible for maintaining your company’s network and systems infrastructure; you know that the days of using the old standbys, ping and traceroute, to manage your infrastructure are long past. “Even a typical mid-sized company network deployment with a 24 port 10/100 router box and a couple of uplinks is going to generate an enormous amount of traffic data. You are dealing with a network that potentially has 100MB to multiple gigabits of data streaming through it. Saving the raw data for the amount of traffic even in a small network and analyzing it by hand is a mind boggling exercise, even if there was sufficient storage capacity or the expertise to figure out the traffic data,” says Debra Deutsch, well-known Network Industry Technologist, and a member of the BBN team that helped create the Internet.
Of the huge variety of tools on the market today, which ones make your life easier and which are not worth the effort? First, we need to define what these tools are and how they are used. Network utilities fall into two basic categories: real time monitoring, and systems analysis tools.
Monitoring tools
Real time monitoring tools are used for detecting events or problems on your network in real time, i.e. as the faults are happening. These tools are typically used by ISPs and enterprises to detect hardware failures, hung systems, and other fatal infrastructure problems. Monitoring tools can be simple or sophisticated, hardware based or software only, but they all tend to be simple to access and use. When you have a bad network link, you do not have time to spend on complex analysis or complicated tools, you want to get your systems back on line quickly. A packet sniffer is a good example – it tells you what traffic is on your network and where it is coming from. If your company is large enough to have a staffed Network Operations Center (NOC) you would want a powerful graphical user interface (GUI) monitoring system.
The advantage to these tools is that they commonly use alarms to instantaneously alert you to problems as they happen. Nevertheless, warns John Eldridge, network engineer and principal at E-Bryonics, “You must be prepared to support very large amounts of performance data for real time access for troubleshooting efforts. The more network components deployed and monitored, the larger the amount of data to manage.” Another disadvantage is that they generally do not keep a history of what was happening during the problem detection. This information is critical for proactive network maintenance and prevention activities.
If all you need is a tool to tell you if your network is down and where the fault is likely to be located, then a monitoring system is probably the right tool for your needs. Monitoring tools available on the market today range from a large number of shareware utilities like Big Brother or mon, to powerful commercial products like Ciscoworks or the HP OpenView toolkit. Most companies do not have the resources or the scale to justify the expense of maintaining a large network monitoring staff, so you will doubtless be interested in one of the more modest tools. If you are the entire IT department, you would probably prefer to have the system page you only when it detects a real problem. Who wants to be disturbed in the middle of the night just to find out that the mail server rebooted?
Although there are many to choose from here are a few particularly useful shareware programs. Big Brother is a set of local clients that test system conditions and the availability of network services. It then sends periodic status reports to one or more DISPLAY servers, or PAGER servers that notify administrators about system problems. Big Brother can be a very useful shareware utility for a LAN network site. It is simple to install and easy to use. The mon website reports, “mon is a general purpose scheduler and alert management tool used for monitoring service availability and triggering alerts upon failure detection. mon was designed to be open and extensible in the sense that it supports arbitrary monitoring facilities and alert methods via a common interface, all of which are easily implemented with programs in C, Perl, shell, etc., SNMP traps, and special mon traps.” While mon is has many great features and a powerful set of extensions, it will clearly work best for the expert network engineer.
Analysis tools
When the facilities design engineers at a GTE site in Irving started complaining that one of their applications was running unacceptably slow, they immediately blamed it on a LAN problem. They wanted to upgrade all their workstations to 100MB Ethernet speed at great company expense. After using some sophisticated infrastructure analysis tools to pinpoint the issue, it was determined that they were attempting to run a large CADD database application in real-time on a server located 1500 miles away. Yes, there were latency problems related to the WAN, but the real cause of the poor performance was that every time someone typed even one character the system would attempt to update the file on the remote server. Upgrading their network connections would have been useless. The ultimate solution was to create a Citrix server on the user community LAN to minimize the number of tiny packets sent halfway across the country.
You want to purchase a new network edge router, but management requires justification for the expense. Your users are complaining about the slow network connection to the West Coast office. How do get the infrastructure information you need to do your job? Using network systems analysis tools, you can gain a better understanding of your network and systems so you are able to answer these questions and solve enterprise systems problems. An entirely new generation of tools is available today, which is an order of magnitude more powerful and easier to use, but they are not for the technologically challenged.
Analysis tools pick up where the monitoring tools leave off. They generally work by filtering through the logs of captured systems data and giving you a picture of possible problems from a historical perspective. “One can use filters to select for specific destinations or sources at the MAC, IP, UDP, or TCP Port numbers to cut down on the analysis processing. Also, tools such as port filtering can be used to mirror in real time the traffic that passes the filters,” comments Deutsch. Although analysis tools can be fundamentally more powerful, they also are frequently more difficult to use because you need to understand the results of the analysis. How useful would the tool be if it informed you of a potential flapping route and you do not know what it was and how to fix it?
Again, analysis tools come in a variety of flavors for all levels of skill and sophistication. Bprobe & cprobe provide measurement of bottleneck and congestion bandwidth using ping. These would be useful if you want to find out if you have enough bandwidth or your users are just complaining on general principles. On the high end of the spectrum is a tool like, Route Explorer by Packet Design CNS, Inc. that lets you visually diagnose and analyze your WAN IP routes over a long period.
Most analysis tools assume that the user would recognize the fault or problem in the data flow by its pattern and characteristics. As a long time user of these analysis systems, Eldridge notes, “In automating the network analysis process, each network component needs to generate specific SNMP MIB data to the network monitoring/management system. Adding the intelligence to the tool to interpret the network statistics is a critical step in improving the value of these systems to assist in identifying performance issues. Statistics often differ depending on the network elements being examined.”
Some gotchas to watch for
Unfortunately none of these tools are truly “plug and play”, so there is a level of skill required to use them properly which can be daunting to the technologically unsophisticated. For proper analysis, accurate data needs to be gathered and must be appropriately organized. “In analyzing network performance issues, a critical step is the creation of an accurate inventory of the various network components of routers, switches, circuits, etc. in order to identify any network component which is causing a performance problem. This inventory must be integrated into any automated analysis tool to increase its troubleshooting value,” says Eldridge.
Deutsch notes, “An important consideration is where on the network you are watching the traffic. If you were doing real time analysis, you would watch at the edge of your network where your traffic goes onto the Internet. That could detect external nefarious activity (i.e. attacking the Pentagon). Remember, not all hacking is directed at external targets, so to detect internal hackers, it may be necessary to monitor multiple internal network nodes as well.”
There are many elements that need to be taken into consideration when you are analyzing a network for faults, the hardware, the network topography itself, the systems and of course the applications. “A lot of this information is dependent on your switch’s ability to handle the data flow to allow you to analysis your traffic. Depending on what you are looking for you might need to start at globally and refine your search as you pinpoint your problem. Important is the ease in which you can on the fly ability to refine what data you are capturing and filtering,” Deutsch continues.
Although the available tools may be excellent, the art of infrastructure analysis is still dependent on the skill and knowledge of the operators. If you are managing your company networks, the best investment you can make is some training in network architecture and analysis. So how do you learn to use the available network analysis tools?
Fortunately, most networking engineering training programs include network analysis modules. Cisco’s extensive and comprehensive offerings include a number of good classes on their tools, so does HP for HP OpenView. If you want to learn more about networks from a more theoretical perspective, I would check if your local college or university has either a certification program or just some network engineering classes. The education you will get will be invaluable for your present job, as well as enhance your future career.
Tools and more tools
From what I saw at the last “Network world + InterOp” show, there are literally hundreds of tools out on the market to help you accomplish this task. Which tools make sense for your IT infrastructure?
At the most basic level you need to decide if you want to look at your systems in real-time or analyze the data after the events. Most companies deploy some combination of both. Which tools you choose is also dependent on the size of your company, the strategic value of your computer resources and the company network topology. For example, every hour that Amazon.com is unavailable to its customers can mean literally millions of dollars of lost revenue. It is critical that Amazon have superb real-time monitoring and excellent preventive maintenance tools. For Amazon, these tools and the staff who know how to use them are strategic for business success. For the majority of companies, something more modest is probably sufficient. The tools available range from ISP quality advanced analysis software to handheld wireless packet analyzers and everything in between. Here is just a small sample of the available tools that I found particularly interesting.
Handheld Wireless Analyzer
Many companies are deploying 802.11 and 802.1x wireless networks because of their lower cost and increased flexibility. AirMagnet Inc., a Mountain View, CA based startup, offers a wireless analyzer with a built in comprehensive suite of wireless troubleshooting tools in a palm-sized Pocket PC. Its robust set of tools quickly helps eliminate connection problems, maintain network performance levels, and ensure a high level of network security. If you have a wireless network, this handy tool is essential.
Visual IP Route Diagnostic Tool
Are you a visually oriented person? Would you like to see your network traffic as a map rather than a text log? Route Explorer by Packet Design CNS, Inc. lets you visually diagnose and analyze your WAN IP routes. You can save up to a year of historical data about your network routes, and create animations of the traffic flow patterns. Many of the big ISPs and telecoms have had similar proprietary tools showing network node maps and histograms for years. This tool allows you to do “what if” scenarios to reduce the risk of network configuration errors, a major problem for many corporations. Although I found the product appealing, it does require that you have an advanced level of knowledge of WAN networking to take full advantage of the features.
Network Flow Analyzer
The most comprehensive suite of analysis products is AppDancer/FA from AppDancer Inc. in Roswell, GA. Their “Network Flow Analyzer” takes a fully integrated approach to infrastructure analysis. By providing fully threaded session analysis, this amazing tool not only monitors your devices, applications and all associated IP flows, but it gives you a view into the real inter-workings of your network. “Because it takes an application centric view of the network, it can help you answer such questions as “‘Is it the applications themselves or the demands or limitations on the devices in my Network?’ or ‘Who is running an unauthorized application using my network?'” according to Tim O’Neill, AppDancer, Director of Sales. This tool does it all! For a mid-sized company that needs sophisticated analysis but has minimal staff resources, this tool could be very useful.
Conclusion
The new network and systems analysis tools can make your life as a system administer much easier. Monitoring tools can quickly notify you of systems and network problems, but they will not give you the history and analysis that you need to prevent future problems. If you willing to take the time to learn, some of the sophisticated analysis tools available today will help you deliver a more reliable and robust network to your customers.
Beth Cohen is president of Luth Computer Specialists, a consulting practice specializing in IT infrastructure for smaller companies. She has been in the trenches supporting company IT infrastructure for over 20 years in a number of different fields including architecture, construction, engineering, software, telecommunications, and research. She is currently writing a book about IT for the small enterprise and pursuing an Information Age MBA from Bentley College.
