REMOTE SITE TROUBLESHOOTING
48 percent of all organizations average longer than half a day to close trouble tickets
Are Network Connectivity Problems a Thing of the Past?
Every year, networks become more reliable. New standards promise interoperability, new devices simplify configuration and advanced monitoring solutions offer to detect problems before users are impacted. IT departments are entering an era where problems will be a thing of the past.
Or are they? A research study of over 300 network professionals in large- and medium-size organizations found that:
- Network professionals spend about 25 percent of their time handling escalations instead of working on key IT projects
- 48 percent of all organizations average longer than half a day to close trouble tickets
- 46 percent of all organizations are under pressure to reduce the time it takes to close trouble tickets
Why is this happening in light of so many IT advances designed to eliminate problems? One explanation is that for every reliability and simplicity advancement, there is an offsetting technology advancement that makes things more complex: unified communications, 802.11ac, more and more Power over Ethernet (PoE) devices, cloud computing or IPv6.
There is also the available toolset and skill level of users to be considered, not to mention the high volume of escalations the network engineering team deals with, pulling them away from important IT projects. Regardless of the reason, there is still much to be gained by improving productivity when it comes to network troubleshooting regardless of media type (i.e. copper, fiber, Wi-Fi), or skill level of the network technician.
To increase productivity and bridge the great divide between the network technician and network engineering, troubleshooting tools must offer automated standardized tests, and the ability to provide actionable visibility when an escalation does occur. The key to overcoming the challenges of remote-site troubleshooting and reducing escalations is to create seamless collaboration between these two groups.
Key challenges of remote site troubleshooting, can you relate?
- Lack of expertise at the remote offices results in travel by senior IT personnel to troubleshoot the problem
- Getting the right resources to the problem by default extends overall resolution time costing the organization money in both downtime and potential travel costs
- Lack of standardized test parameters and documentation makes problem triage more difficult and time consuming to ultimately resolve
A good network connectivity tester will pay for itself within just a few months with the time it saves doing troubleshooting and validation testing.
This white paper will discuss strategies that your organization should consider when looking for ways to help the on-site network technician become more capable of identifying and resolving problems on their own. When an escalation does occur, nonetheless, make sure it’s done with actionable visibility such that network engineering knows where to begin problem triage, which may include taking over troubleshooting at the remote location via remote control. Sound intriguing, read on.
Where to begin? We can’t stress this enough, start by planning ahead.
A little planning ahead can help you to develop a solid strategy that will save time in the long run. Planning ahead will also help you with budget planning for the proper tools to get the job done as efficiently as possible. Here are some questions you should ask to start planning your strategy:
Step 1: Evaluate the on-site support situation
- Is there any IT staff on-site?
- If so, what is their expertise level?
- Do you have critical and non-critical sites?
- Getting the right resources to the problem by default extends overall resolution time costing the organization money in both downtime and potential travel costs.
Site criticality may require different remote-site troubleshooting strategies based on budgets and on-site staff skill level. Once you have had the chance to think about your remote-site staffing situation, then you need to think about what you need to have visibility of, to understand and potentially resolve the issue by things you can do from headquarters, such as reconfiguring a switch. Alternatively, you might be able to identify an issue that you can give direction to the on-site team to resolve or take over remote-control of test equipment at the remote site, eliminating the need to take time out to travel to the site yourself and possibly even help teach the on-site technician as they watch and learn.
Before traveling to the remote site, you should be able to have the following facts validated by the on-site technician:
- What is the switch slot/port/VLAN?
- Is there a problem with the physical cable?
- Are all users affected or just a select group?
- Are network services obtainable from a given wall jack or access point?
- Is a power-enabled device getting proper PoE voltage at the wall jack?
With decades of experience working with IT pros at all levels and different organization types and sizes, we’ve worked together to develop methodologies that will simplify and shorten the entire problem-solving process. These are the top three considerations we recommend to help you take control over remote-site troubleshooting challenges:
- Automated Testing
- Simplified Troubleshooting Across Media Types
Different Ways to Tackle Remote Troubleshooting
So the good news is that there are different ways to tackle the challenge of remote-site troubleshooting based on what the on-site support personnel profile looks like, how you defined your requirements in the planning phase and then there is of course budget considerations.
Even with no IT support on-site or lower-skilled technicians, we’ve worked hard to develop different methods through simple workflows and remote control that will help you to get the job done.
Step 1: Automated Testing
There are several benefits to automated testing. First, it’s much faster than a typical trial-and-error test. Second, it’s more thorough than a manual approach, which means it can find problems that the technician may not have even considered. Third, it allows anyone, regardless of skill level, to run tests and identify problems. Lastly, it creates a standardized process and workflow, which will be useful if an escalation does occur. The escalation team doesn’t have to start from scratch. There is a standard set of tests already run that can be referenced to identify the problem domain so triage can begin immediately at the failure point.
Step 2: Simplified Troubleshooting
While the AutoTest uncovers a wide variety of problems on its own, not all problems can be found using this method so the tools should also be flexible enough to provide additional diagnostic capabilities.
AutoTest — Even if the initial AutoTest does not identify the problem, all the measurement results are readily available to help the technician understand what is happening. Furthermore, the AutoTest can be modified in seconds and re-ran to test a different server, application or wireless connection.
Wired Tests — Sometimes the physical layer is the culprit so it’s important to be able to verify the cable, PoE, the nearest switch, VLAN, network services, and even the ability to inspect the fiber-optic connectors for contamination.
Wireless Tests — It’s important to be able to see all the networks, AP’s and clients in range to quickly identify problems such as improper security, interference, bandwidth hogs, overloaded channels, unauthorized devices and more.
Step 3: Collaboration
Simply stated for the purpose of this discussion, collaboration is focused on improving the workflow and information exchange between the technician and network engineering for the purpose of resolving problems faster as well as managing adds, moves and changes more efficiently.
- Have a centralized results database for both wired and Wi-Fi results that houses data that cross-functional IT teams can reference for the job at hand
- Allow senior IT staff to troubleshoot wired and Wi-Fi issues remotely using tools located at the remote site
- Allow senior IT staff to see the physical environment a technician is working in, using tools located at the remote site
- Simple in-line packet capture for solving more in-depth problems
Now let’s dive into these considerations in more detail to help you realize new ways of tackling problems you may not realize is right at your fingertips.
Centralized Results Database — A centralized results database for both wired and Wi-Fi results will allow cross-functional IT teams to reference a single source of results data to enable better understanding and discussion of test results. This comprehensive, searchable database allows you to easily identify changes in the network and see who is testing what, when and where.
There is no space limit, no time limit, and no need to retrieve and transfer data from testing devices to storage at the end of the day. Link-Live is available to all Handheld Network Tools: LinkSprinter, LinkRunner AT, OneTouch AT and the AirCheck G2. If you don’t have connectivity at a drop, the testers can buffer results, which are pushed to the Link-Live Cloud Service when connectivity returns.
Link-Live Cloud service is a centralized management, collaboration, and archival workspace for network connectivity test results from NETSCOUT's handheld network testers.
Web Remote Interface — While it’s not always possible to get someone with the right skill level to solve a problem at the location where the issue is occurring, the OneTouch AT and AirCheck G2 can be accessed and controlled remotely through Link-Live. Not only can the remote user see what the technician is seeing, but they can control the tester to perform remote troubleshooting. This also serves as a way to teach the on-site technician as they watch how network engineering uses the tools to identify issues and solve problems.
Camera — Connect a webcam to the OneTouch AT USB port and the remote helper can see live video of the physical environment the technician is working in. This is useful if the technician is in a wiring closet or a data center and the remote colleague needs to see the switch or patch panel, for example.
In-line Packet Capture — A trace file is crucial to have for very difficult problems or as evidence to provide to an outside group, such as application developers, service providers or equipment suppliers. Collecting this information typically requires reconfiguration of the switch or a network tap. This can take 30 minutes or more. Worse, many technicians may not have access to switch provisioning or a tap. That means even more delay as the problem is escalated to another individual.
The OneTouch AT can perform an inline packet capture in just a few screen touches without the need to access the switch or a tap. This means the technician can capture the problem packets immediately while the user demonstrates the problem.
Establishing and ROI
To help you understand and justify the investment in tools that will help your organization become more efficient and troubleshoot problems at remote sites, we’ll select our most full-featured handheld tester, the OneTouch AT, to look at the time saved during key troubleshooting processes. You can imagine the ROI will be even better when considering other tools in our handheld tester lineup. Functionality such as:
Automated Testing — Table 1 compares the amount of time it would take to perform the AutoTest functions with the actual time of the AutoTest. The time is dependent on the skill of the technician as well as how many applications would need to be tested.
Troubleshooting — It’s less straightforward to quantify these savings, as it is highly dependent on the actual problem and the skill of the technician. Users of other NETSCOUT Handheld Network Test Solutions generally report 30 to 40 percent faster troubleshooting, but we will set that aside and consider it a “bonus” savings in addition to what is demonstrated here.
Collaboration — To quantify the time savings in these scenarios, we compared time to set up a packet capture using port mirroring (roughly 20 minutes) versus the inline packet capture with the OneTouch AT (three minutes). In instances where the technician does not have access to reconfigure the switch, the time savings would be much greater.
Better collaboration offers a bigger benefit by reducing the overall time to close trouble tickets. Without the OneTouch AT, it’s often difficult to get all the relevant data in front of the appropriate people quickly. This is a significant cause of trouble tickets dragging on, sometimes for days. Armed with the OneTouch AT, the first responder easily generates a report or a trace file and shares these plus provides remote access for the rest of the team in real-time. So while the total amount of time the staff spends on a problem may not be reduced, the time the end-user spends waiting for a resolution is greatly reduced.
Again, such a benefit is hard to quantify — so we won’t — but for many organizations this may be more valuable than actual hours saved for the department staff.
Below in Table 1 is an estimation of the savings expected by using handheld cable and network connectivity testing solutions. Even ignoring the time saved in solo troubleshooting and purchasing of the top of the line model, one would expect payback in less than six months.
|Trouble tickets per month per tech||20 (median)|
|Minutes per ticket||90|
|AutoTest time||1 minute|
|Time to manually perform AutoTest functions||60 minutes|
|Time saved per ticket||59 minutes|
|Hours saved per month||19.7 per tech|
|Percentage requiring packet capture||19% (average)|
|Number requiring packet capture||3.8|
|Time for packet capture setup||20 minutes|
|Packet capture setup with OneTouch AT||3 minutes|
|Time saved per capture||17 minutes|
|Hours saved per month||1.1 per tech|
|Number of users||2 per OneTouch AT|
|Total time saved||41 hours per month|
|Total monthly savings||$2,489|
|OneTouch AT cost||$10,000|
Table 1: An estimation of time and cost savings provided by using handheld test solutions.
Appendix A – An Hour of Troubleshooting in One Minute
Getting to the root cause is critical but can involve extra steps. For example, when you go to the doctor’s office, regardless of your complaint, you’re first seen by a nurse who immediately measures your weight, temperature, blood pressure and often reviews your health history. This step not only saves the doctor’s time, but often catches issues that might otherwise be overlooked.
This concept also applies to testing a network. Since your team doesn’t usually include a nurse, all NETSCOUT handheld testers automate a complete test of “network vital signs” as part of an AutoTest. This can take what is traditionally a range of 30 to 60 minutes and compress it from less than 10 seconds to about a minute, depending on what needs to be tested and which NETSCOUT handheld tester you use. The results are then compared against user-defined limits to provide a simple pass/fail result. This approach not only saves time, but allows technicians to solve more problems.
Here’s an example of a full wired/Wi-Fi test using the OneTouch AT:
|Testing Step||OneTouch Network
|Basic Connectivity (Wired or Wi-Fi)||1 minute||Cable Tester, PC, Wi-Fi Utilities||5 minutes|
|Infrastructure Services||PC, Utilities||5 minutes|
|Wireless Operation and Performance||Two PC’s, iPerf||10 minutes|
|Network Services and Application Performance||Packet Capture, Protocol Analyzer||40 minutes
Table 2: The OneTouch AutoTest performs nearly an hour of manual testing in one minute.
NETSCOUT Handheld Network Tester lineup
Here’s a quick line-up of the various products with a high-level overview of functionality. Most IT organizations find that a mix of tools provides the best ROI and efficiency gains based on the site criticality, media type requirements, and the skill level of users.
Also, keep in mind that many of these are sold as kits or in multi-unit packs so you can get more bang for your buck when making the purchase.
Solutions from NETSCOUT
LinkSprinter Network Tester
Test and Validate Network Connectivity in less than 10 seconds
The LinkSprinter network tester provides network connectivity troubleshooting and identifies error problem domain in less than 10 seconds. Automated reporting to the included Link-Live Cloud Service ensures network connectivity status and path information is documented for every link.
The LinkSprinter gives you 5 essential network tests in the palm of your hand:
- Power Over Ethernet (PoE) Tester
- Link to the Switch
- DHCP Connection
- Gateway Connection
- Internet Connection
More information at:
LinkRunner AT Network Auto-Tester
Quick and complete copper and fiber network connectivity testing
The LinkRunner AT Network Auto-Tester performs 6 essential network connectivity tests in 10 seconds, enabling you to quickly and accurately identify and solve network connectivity problems. Cloud reporting automatically uploads network connectivity test results to the Link-Live Cloud Service for reporting and project management.
- Powers on in 3 seconds, 10-second AutoTest
- Verify connections at 10M/100M/1G on copper and fiber
- Cable testing and verification
- Nearest switch, port and VLAN information
- Supports PoE testing and IPv6
- Ping and TCP port connectivity tests to up to 10 user-selected targets
- Stores up to 50 test results
More information at:
OneTouch AT Network Assistant
Client-to-Cloud troubleshooting in 60 seconds
The OneTouch AT Network Assistant greatly reduces troubleshooting time through a streamlined, three-step approach:
- The unique AutoTest replaces multiple tools and an hour of troubleshooting time
- A powerful set of network performance measurements to troubleshoot wired and Wi-Fi networks
- It enhances team collaboration through a simple web-remote interface and easy-to-use inline packet capture capabilities
More information at:
AirCheck G2 Wireless Tester
Test and Validate Network Connectivity
The AirCheck G2 provides fast, simple, and accurate isolation and troubleshooting, thereby reducing the time to resolve wireless issues. AirCheck G2 simplifies wireless troubleshooting by providing:
- Support for the latest Wi-Fi technologies (802.11a/b/g/n/ac)
- A one-button AutoTest
- An instant view of test results including network availability, connectivity, utilization, security settings, rogue hunting, and interference detection
- Connectivity to a centralized test results management platform
More information at: