Troubleshooting Ethernet Services with Packet-Capture and Decode Capabilities—Part 1

Ethernet networks have experienced incredible growth in the amount of data that they carry. With increased reliability, Ethernet networks have transitioned from being pure Ethernet data pipes to carrying Ethernet services. Ethernet commercial services now carry triple-play services for enterprise and commercial customers, while mobile backhaul networks now carry time-sensitive and mission-critical data services across packet networks, enabling mobile convergence. As the service offering becomes more complex, network engineers and field technicians are required to respond to more troubleshooting service calls that require them to rapidly pinpoint, analyze and report quality of service issues.

Network troubleshooting involves performing a number of complex procedures to identify where and why a network failure is occurring. While network technicians and engineers usually have very little information about the event, and they must search through multiple possible causes of failures. What’s more, the task becomes even more difficult with the pressure of having limited investigation time, in addition to knowing that customer’s may be greatly affected.

Among the many tools available to the technician, a very popular one is the ability to capture the traffic on the affected circuit and to decode it. Decoding the traffic usually refers to the interpretation of the content of the header to identify any issues in the content of the header, such as modifications and incorrect content. Decoding also allows investigators to identify the true content of the circuit as all the traffic, such as customer traffic and network command-control traffic, is captured. A technician can then search through the list of packets and identify out-of-place or inconsistent traffic by analyzing the overhead content of the captured traffic.

This article offers insight into troubleshooting Ethernet services with the implementation of EXFO’s packet-capture and decoding capabilities.

Field Applications

Although portable test units can provide powerful test capabilities, there are a few situations where packet capture can provide more information for enhanced troubleshooting. Here are a few examples:

Top-Talker Analysis

A typical issue with network operators is identifying those who monopolize the bandwidth, i.e., stations that consume too much bandwidth and users that engage in heavy download, illegal streaming or even virus-infected computers. From a carrier perspective, the monopolization of bandwidth can indicate congestion or incorrect configuration of transport devices.

A packet-capture session can complement troubleshooting by capturing the actual content of the pipe as the issue occurs. Offline analysis can then provide bandwidth and utilization statistics, such as top talkers (MAC, VLAN and IP) or packet distribution.

• An analysis tool can provide the statistics per conversation and identify those who monopolize the bandwidth
• Top-talker analysis is performed by sorting the analyzed statistics

Performing Deep-Packet Inspection

The three main configuration issues with packet-forwarding devices is fragmentation of the traffic, VLAN tunneling and the header fields overwrite:

• Traffic fragmentation occurs when a device must segment the traffic into smaller sizes in order to transmit within the maximum transmission-unit limit of a pipe, i.e., the maximum data size allowed in a pipe. When fragmentation occurs, performance is typically reduced as less effective bandwidth is available with the increased frame numbers required to transmit the same data size.
• Tunneling occurs typically when an Ethernet device adds, swaps or removes VLAN tags as it processes traffic. Tunneling typically occurs at the edge of the network where untagged traffic is first tagged then forwarded on to the L2 network or when tagged traffic is untagged and forwarded to the proper destination.
• Header overwrite sometimes occurs when traffic with a specific priority, usually IP TOS/Diffserv, is forwarded with a different priority, causing QoS issues during congestion.

In these three cases, capture provides a real-time view of the issue and allows network investigators to perform deep-packet inspection using the overhead decode capabilities.

Troubleshooting the Customer’s TCP Issues

Although EXFO’s TCP application provides significant service turn-up testing of TCP pipes, troubleshooting the customer’s TCP issue requires testing on the actual customer’s TCP streams to identify overhead or connection issues.

The packet-capture capability complements TCP testing by providing capture and decoding analysis of the customer’s actual TCP exchange. Decoding provides insight into the content of the overhead and provides further sequence analysis, such as providing a graphical flow of the exchange and identifying retransmission and duplicate acknowledgements. By using a decode capability, the user can therefore analyze the events leading to the duplicate or retransmission and also ensure that the proper retransmission has occurred after the event.

New Services Analysis

In addition to VoIP and video (IPTV), carriers are now implementing Ethernet-synchronization services, especially for mobile backhaul. Precise Time Protocol (IEEE1588 PTP) is a new protocol designed to establish and maintain synchronization in a packet network based on a client-server architecture, where client boundary clocks maintain clock synchronization and stability using notification and requests from a master clock. Packet-capture and decode capabilities provide a simple and intuitive approach by enabling the capability to capture these services and perform conversational and deep-packet analysis, identifying conversation issues, such as packet loss and incorrect sequences.

• An example of a PTPv2 packet capture sequence.
• An analysis tool can provide deep-packet decoding and conversation analysis, which can determine issues in the flow of traffic between boundary clocks and grand master clocks.

Current Packet-Capture Alternatives

The traditional method for performing packet capture and decoding required extensive hardware and lab equipment to handle the large amount of data and the high packet-rate. As processing power increased, desktops and laptops became viable platforms for simple capture and decode—with dedicated hardware remaining the ideal tools for lab and process-intensive applications. However, as desktop and laptop computers simplified and made capturing more affordable, along came a number of drawbacks:

• Using laptops also increases the amount of equipment that a technician must bring to a job site. As technicians become more mobile, controlling the quantity of equipment that is moved from site to site becomes an issue.
• In an effort to control costs and operating expenses, network operators have reduced testing and troubleshooting budgets to minimum levels, this is why having a dedicated computer for capture and decode functions becomes more and more difficult to justify.
• Desktop and laptop computers do not usually provide gigabit-rate capabilities and costly adapters, switches or optical/electrical converters must be associated to the capturing device when testing must be performed over gigabit rates or optical connections. Adding these devices increases the complexity of the test architecture and adds extra points-of-failure—something that should be avoided when performing troubleshooting with limited information.
• As 10GigE links are becoming more popular, engineers are now starting to troubleshoot aggregation points that use 10GigE links. With a typical computer or laptop scenario, 10GigE capabilities are very costly adapters and are usually performed by using a converting switch.

Part 2 of this article will present EXFO’s packet-capture and decode capabilities as well as extra features that increase the efficiency of the test cycle.