VanRan Communications helps customers deploy converged voice and data systems based on AVAYA technology. When problems emerge with voice quality or the stability of a system, engineers need to quickly isolate the issue and identify if the network or hardware is to blame. Unfortunately, this process is time consuming and difficult, often taking months of troubleshooting time with tools like Wireshark. Making matters worse, issues are not always isolated, causing VanRan to redeploy additional resources and hardware in an attempt to fix the problem. To accelerate the troubleshooting process and increase accuracy, VanRan engineers needed a solution that simplified simultaneous data capture and instantly combined data flows so engineers could accurately isolate a problem, prove its source, and fix it fast.
VanRan selected NetScout Network Time Machine™, an all-in-one packet capture appliance for network and application analysis. With the solution, VanRan engineers have dramatically improved their ability to identify root cause, increasing their success rate from roughly 50 percent to more than 95 percent. Furthermore, engineers have dramatically reduced the amount of time spent finding a problem by approximately 50 percent. The ability to streamline the troubleshooting process has helped increase customer satisfaction and allowed VanRan engineers to get more done in less time.
Network Time Machine™
“It’s quite simple. Our customers rely heavily on their voice and data systems. When problems emerge, our first order of business is to find the root cause. If it’s not our equipment, we have to be able to prove to the customer the true source of the issue so it can be resolved as quickly as possible. Basic network protocol analyzers just can’t provide us the level of insight and the level of assurance around root cause that our business requires and our customers expect. With Network Time Machine, we have the confidence that we can find and fix any problems fast.” - Ron Bohannon, Chief Technology Officer, VanRan Communications
VanRan is a full-service, converged voice and data telecommunications company specializing in voice, data, IP, network services and wiring. The company offers complete systems, parts, upgrades, installations and maintenance for customers, and has the ability to analyze, recommend and implement turnkey, end-to-end communications solutions. VanRan has been an Authorized Avaya Business Partner since 1992 and an Authorized AT&T Solutions Provider partner since 1995. The company serves a variety of organizations from small businesses to large enterprise, including many of the Fortune 500, and a variety of state and local government agencies.Challenges
VanRan specializes in selling, installing, maintaining and troubleshooting voice networks. Their customers often have complex multi-site deployments and when a problem emerges – such as poor voice quality, one-way voice calls or call completion issues – the VanRan engineering team’s first order of business is to isolate the problem and prove whether it is a network or hardware issue. Unfortunately, the process of hunting and digging can sometimes take months with traditional troubleshooting tools like Wireshark.
“Typically we need to look at multiple places in a customer network to properly troubleshoot. This means at least two network locations, and two to four points at each network location. Not only does that create a lot of travel for our engineers, but also when using traditional troubleshooting tools, it generates a lot of data that we then have to manually correlate in an attempt to find the issue. To be honest, it’s a complex and time-consuming process that doesn’t always produce the results we want,” said Ron Bohannon, chief technology officer, VanRan Communications.
As a matter of fact, VanRan engineers can often spend months troubleshooting a problem and sometimes never truly isolate the issue. When this happens, the team is forced to swap hardware units and modify configurations in an attempt to find a workaround to solve the problem.
“When we’re asked to troubleshoot, more often than not, it’s due to some other issue on the network that is impacting the voice system. In these cases, we have to prove to the client that it’s not the voice system. This is no easy feat. It means providing detailed information, sometimes even at the packet level. To get this granular information, we have to capture a lot of data and analyze the flows of information to root out the problem,” Bohannon said.
Unfortunately, today’s traditional tools do not allow VanRan to simultaneously capture multiple stream of information with correlated time stamps, or allow the team to go back in time and dig into actual packets for specific details. Therefore, engineers spend a lot of time filtering through information and doing manual analysis. And in the end, the team still may not be able to identify root cause because the traditional tools only provide basic packet information.
“It’s quite simple. Our customers rely heavily on their voice and data systems. When problems emerge, our first order of business is to find the root cause. If it’s not our equipment, we have to be able to prove to the customer the true source of the issue so it can be resolved as quickly as possible. Basic network protocol analyzers just can’t provide us the level of insight and the level of assurance around root cause that our business requires and our customers expect. We end up wasting time on travel and analysis simply because we don’t have a sophisticated solution that streamlines the troubleshooting process,” Bohannon said.Solution
To solve its VoIP network troubleshooting challenges, VanRan selected NetScout Network Time Machine.
“We had some experience with other NetScout products, so when the sales rep visited us and demoed the Network Time Machine, it was an instant hit with the team. We moved to quickly order two units, knowing we could streamline the troubleshooting process and better meet the needs of our customers,” Bohannon said.
As an all-in-one solution for back-in-time root cause analysis of network and application problems, Network Time Machine delivers 100 percent packet capture, so VanRan engineers never miss a key event. Its innovative Performance Bottleneck Analysis (PBA) visually identifies if a problem is on the server or the network, and the application-centric analysis finds problems without the team needing to look at a high volume of packets. Advanced data mining quickly pinpoints the relevant packets in multiple terabytes of captured data, saving engineers valuable time.
All of this functionality is integrated into a single plug-and-play appliance that helps VanRan reduce the initial expense, set-up time and ongoing maintenance associated with Network Time Machine. Furthermore, once plugged in, it automatically identifies applications, analyzes performance, and displays relevant statistics in user-configurable dashboards. And, when needing to correlate information from multiple locations, VanRan engineers can use the auto-multisegment analysis capability for collaborative packet capture, retrieval and analysis across multiple Network Time Machines.
“NTM is a sophisticated tool that helps our engineering team shine in the field. We currently have two units, but that’s only because the two units help us cycle through the workload much faster then before. As demand grows, so will our units,” Bohannon said.
Since putting Network Time Machine into service in February of 2012, Bohannon estimates the success rate of his team identifying and proving the root cause of a problem has increased from 50 percent to more than 95 percent.
“Before Network Time Machine, our ability to find the true root cause of an issue using traditional protocol analyzers was difficult. Our engineers were often forced to swap out hardware and make changes until the problem was resolved. Since none of our customers are local, we’d be traveling to sites multiple times just to troubleshoot. Looking back, the logistics were crazy, but we were doing the best we could with the technology available,” Bohannon said.
With Network Time Machine, not only is the VanRan team confident they can isolate root cause, but the time it takes to complete the troubleshooting process has been cut in half.
“Our entire process for troubleshooting problems has changed. We used to have tremendous admin time associated with travel and gathering data, just to get to the point of analysis,” Bohannon said. “Now with Network Time Machine, we can ship the product, the customer can plug it in, then we can remotely access the unit for configuration. If it’s a voice quality issue, we wait for data collection, then evaluate and fix the problem. If it’s a signaling issue, we work with someone at the client location, but we can still dramatically streamline the process.”
While VanRan offers ongoing support for its customers, the company is often brought in to identify problems at new companies with existing VoIP deployments. These customers often work for months to identify the cause of a problem, bringing VanRan in once internal engineers are unable to resolve the issue. In either case, proving the root cause of a problem is key and providing evidence to the client is often a requirement for getting approval to fix and complete the project.
“This is where Network Time Machine really shines. It allows our engineers to prove, without question, the root cause of a problem. For example, recently we had a customer with a one-way voice call issue. The call would complete, but the person on the other end could not be heard until they pressed a button on the handset,” Bohannon said. “In this case, we suspected it was a firewall problem, but we needed to prove it to the customer before we could move ahead and fix it. Network Time Machine allowed us to quickly record the packets coming in and out of the firewall. Once we presented the data that showed the actual change to the packet – before and after – the customer approved the fix and we moved on.”
VanRan also troubleshoots a variety of call quality issues. Having the ability to dive deep into packet information allows engineers to find root cause, even in situations where a MOS (Mean Opinion Score) is high. “We had a situation where the MOS was excellent, but when we listened to the call on playback through Network Time Machine, it was evident that there really was a problem with quality. When we looked into the packet level information, we found a duplex issue, and the voice sample was being blocked and transmitted back to the user too late. Without Network Time Machine, we couldn’t have identified the anomaly and isolated root cause as fast as we did,” Bohannon said.
Network Time Machine gives the VanRan engineering team a new level of confidence when troubleshooting customer problems. It has allowed the company to dramatically reduce the resource time needed to complete a project, and optimize workloads so Bohannon can make the troubleshooting service a reliable source of revenue.
“I would rather have my engineering team using Network Time Machine versus any other troubleshooting solution. It has kept me from having to hire another engineer, essentially paying for itself already, and has allowed my team to get more projects done in less time. Troubleshooting is now a revenue generator, instead of an unpredictable black hole for productivity,” Bohannon said. “Network Time Machine provides a new level of accuracy for our team when diagnosing VoIP issues. When customers call us, they know we’ll restore quality to their phone systems and do it fast. We couldn’t make that promise before we had Network Time Machine.”