Validating 10G Network Performance
Table of Contents
These are just a few of the benefits brought to the modern data center by 10Gb links. Using high speed connections, network engineers create a pipeline which supports critical traffic such as business applications, voice and video services, server backups, and Unified Communications. These 10G links are expected to be the under-pinning for a superior user experience, despite the daily-increasing load. This would seem reasonable, given that 10G connections can support a tremendous amount of data. However, since these links carry the most critical and demanding traffic generated by the business, it is surprising that in most environments they are installed with little or no validation. Engineers simply install the switches, connect the infrastructure cabling, wait for the link light to appear, and assume the overall performance of the network and applications will automatically improve.
This “Plug and Play” method of 10G link installation does not guarantee high performance in the data center. In fact, it may leave you with questions, especially when a problem arises or performance does not improve. These questions include:
- Does the link light really guarantee 10Gbps?
- Can the switch fabric handle 10Gbps end to end?
- If not, why? Where is the problem?
- Will QoS or VLAN traffic affect throughput levels?
- Are we getting the performance we expect out of all these expensive links?
The Challenge of Testing 10G Connections
In environments where network engineers attempt to validate and test 10G infrastructures, most find that they simply do not have the tools to properly do so. One reason is that the bandwidth measurement tools available to them are often software-based, and are installed on laptops for easy deployment. These tests are run by connecting a laptop on each end of the network path and generating traffic between them. However, when testing 10G network paths, software-based test tools present two major problems:
- A laptop typically cannot connect to a 10G link
- Even if it could, the IP stack and internal drivers running the NIC cannot support true line-rate traffic generation or capture
These problems make 10G validation with software tools unreliable. When the throughput test fails, it is difficult to know if the high speed link really had a problem, or the low result was due to the inability of the tool to achieve and maintain line-rate.
Some IT organizations have recognized this problem and have opted to purchase hardware based tools which can achieve a line-rate 10G test for network validation. These tools have purpose-built 10G interface cards which were designed to handle a 10G load. They do well at validating the network performance and can accurately measure bandwidth and loss using a variety of traffic settings. However, most of these tools (which are typically designed for telco/service provider field technicians) do not provide the network visibility necessary to perform root-cause analysis on a poor test result which limits their use after an initial link validation. Engineers may experience a failure on the performance test, but are limited in using the tool to find the root problem on the network.
10G Network Performance Test with Path Visibility
To effectively validate a 10G path, and the network performance in general, we need to have visibility into more than throughput. As discussed above, a hardware based tool can validate whether a connection can support 10Gbps at a sustained line-rate. After the initial validation test on new connections, these tools do little to troubleshoot a failed result or future network performance problems, which can lead to them making a permanent home on a storage shelf. To prevent this, the performance measurement tool also needs the capability of detecting problems such as cable faults, switch interface problems, misconfigurations, buffer drops, or high switch CPU utilization which cause poor performance.
The Network Performance Test (NPT) available on the NETSCOUT OptiView® XG Network Analysis Tablet enables network engineers both to measure the performance of network connections at speeds up to 10G while visually isolating the problems affecting the test results. Test traffic can be configured with several settings, including the addition of VLAN tags and QoS metrics, which stress 10G links and the underlying switching fabric. The NPT feature can run as a full 10G test in the data center between two OptiView XG units, which stresses switching fabrics, both physical and virtual. Multi-endpoint mode allows for the test to be run against up to eight remote endpoints distributed throughout the enterprise, measuring critical links connecting users to the applications required to run the business.
If any part of the NPT test fails, the Path Analysis feature on the XG can be used to isolate the root cause along the network path that is responsible for the problem. Anything from an interface that is dropping packets, an excessively utilized link, a switch with a high CPU, or a faulty cable can be visually isolated and resolved.
When you don’t get what you paid for
After experiencing performance issues between key sites, the network architects for a major US city decided to upgrade their metro area link from 1 Gig to 10 Gig Ethernet. Despite claims of innocence from their service provider, the link was not delivering the expected performance improvements. Using the OptiView XG to validate the 10G link, the city’s network engineers were able to determine that the link was delivering 7 Gig upstream and 1 Gig downstream, and that the configuration problem was not on the city’s network. Confronted with the evidence, the service provider rechecked configurations and corrected the issue.
What Settings to Use
Many Network Engineers struggle with what settings they should enter when performing SLA type testing. Every network has different characteristics such as latency, load, packet loss, etc. Not to mention that these variables may change at certain time of the day or during different business cycles.
As part of the NPT test procedure, a configuration test runs prior to the actual performance test. The configuration test will validate that the services are configured as intended and traffic flow can be established between the source OptiView XG analyzer and its peers and reflectors.
Roundtrip testing at 1G with a “reflector”, bi-directional testing at 1G/10G with a “peer”
If all of the configuration tests do not pass, you can review which parameter caused it to fail, what the recorded value actually was and you can make your adjustments accordingly.
If all of the configuration tests pass, the performance test runs. The performance test quantifies network performance over a time period that you select on the Performance Test Suite Configuration screen.
Measuring 10G Links in the Data Center
Let's look at an example of how to validate a 10G link in the Data Center. First, an OptiView XG must be connected at each end of the network under test. This is to ensure that the test points on each end can both transmit and receive at 10Gbps, as well as to ensure the hardware is capable of handling the timing and traffic features of the test.
Using the NPT configuration settings, the traffic levels for the test can be configured to run for any length of time between one minute and 24 hours. Traffic configuration settings can be adjusted to measure network performance for key applications and services. After starting the Network Performance Test, the XG will automatically generate a report with the test results, which can be archived for documentation.
The NPT feature also supports multi-endpoint mode, which will allow up to eight simultaneous performance tests at once. Remote endpoints in the form of a LinkRunner AT or OneTouch AT can be connected in distributed locations throughout the network for validation of several key network paths with one test. This allows for key pathways to be measured for throughput, jitter, packet loss, and overall performance. When the test reveals problems, the visual Path Analysis feature of the OptiView can be quickly run to detect the underlying network issues that impact performance.
The traffic levels generated by business critical applications and services are increasing day by day. To support this traffic at a high level of performance, high speed network links are being installed in data centers and in core connections of the network infrastructure. A link light is not a positive test for a 10G link, and should not be used as a validation test. With the OptiView XG, these critical connections can be tested for bandwidth, jitter, packet loss, and more, while using several different traffic configurations to correctly stress test the whole 10G path. The test can be run across one 10G link, or on several paths using multi-endpoint mode. When a test does fail, the OptiView XG provides the visibility necessary to quickly find root cause using Visual Path Analysis, which eliminates guesswork and saves time in troubleshooting these key components of the network infrastructure.