Using Linux to verify DSCP

As previously demonstrated on Windows, here we’ll verify DSCP values using Linux. When implementing application(s) that require QoS such as voice and video, it is critical to verify the integrity of DSCP values. Keep in mind there are many applications, including network devices (such as control plane protocols OSPF and VRRP) that utilize the DSCP field. In this demonstration, we’ll verify DSCP value 46 is being preserved using the Linux terminal and protocol analyzers TCPDump, TShark, and WireShark for CLI/GUI verification.

Keep in mind there are many ways to verify a given value in a PDU, in this case an IPv4 packet header. Since the Internet is full of useful information, we’ll keep a narrow focus by only displaying test traffic that we’re generating. Again, in this case a particular DSCP value.

When using TCPDump or TShark you’ll need to open two terminals, one to generate the traffic and the other to verify it. Once opening the first terminal, enter the ping or man ping (displays the user manual) command.

Linux DSCP Ping Info

Linux DSCP Ping Man

Per the options, we’ll use the -Q switch to mark the ToS value. Remember the ToS field in an IP header is 1 byte (or 8-bits) and DSCP refers to the first 6-bits in the ToS byte. If you do not yet understand binary it’s helpful to know that multiplying a given DSCP value by 4 gives you the corresponding ToS value. When converting ToS to DSCP, simply take the ToS value and divide by 4. Below are some examples converting DSCP and ToS values:

DSCP 26 (011010) <==> ToS 104 (01101000) DSCP 46 (101110) <==> ToS 184 (10111000) DSCP 48 (110000) <==> ToS 192 (11000000)

Once you understand this conversion, it becomes easy to generate the traffic! Now, let’s open a terminal and ping a node to test DSCP values. In Linux the ping command defaults to continuous mode so we can let this test run in the background but remember to terminate it once your finished (Ctrl+C).


Linux Ping with ToS Switch

Now that we are generating traffic (ICMP) marked with a ToS/DSCP of 184/46, let’s verify DSCP is preserved using some common protocol analyzers. The following examples focus on displaying ONLY traffic of interest using protocol filter proto[expr:size] and logical operator AND. The main reason for this approach is the local NIC may be sending/receiving hundreds or thousands of PDUs per a second, thus a protocol analyzer without refinement can easily provide too much information! Also, by incorporating protocol filters, this reinforces your knowledge of a given protocol. Remember, there are many tools and techniques to analyze the same traffic so the examples provided simply demonstrate a few different ways to verify DSCP values using Linux.

TCPDump

TCPDump is a protocol analyzer that runs in the CLI and like WireShark, uses the pcap file format. While the examples below rely on the CLI, you could also pipe the TCPDump output to a file and open it with WireShark. In the following TCPDump examples, we use the following switches:

     -i indicates which interface to capture traffic
     -v increases default verbose output (range -v to -vvv)  
     ip[1] examines second byte (ToS field) in the IP header
     and logical operator, both protocol filters must match
     icmp[0] examines first byte (Type field) in the ICMP header

The first screen below, uses protocol filters ip[1]==184 and icmp[0]==8, so only ICMP ECHO Requests (Type 8) with ToS/DSCP of 184/46 is dispalyed. The filter matches verify ICMP traffic, marked with ToS/DSCP 184/46 is leaving the local NIC (linux.local) with a destination of 10.0.0.54. Use of logical operator and requires BOTH protocol filters to match. If operator or had been used you might see ICMP traffic regardless of ToS/DSCP value or non-ICMP traffic marked with ToS/DSCP 184/46.

TCPDump ICMP Echo Request Filter
TCPDump Echo Reply

Notice the TCPDump output displays ToS values in HEX, in this case 184 translates to b8. If DSCP was displayed, the HEX value would be 2e (DSCP 46). You can easily find HEX converters online so this is just another way to visually verify what we are testing.

Now, stop TCPDump (Ctrl+C) and change protocol filer to look for ICMP ECHO Replies (Type 0). If TCPDump displays filter matches, this confirms DSCP values are being preserved across the network, between source and destination nodes.

TCPDump ICMP Echo Reply Filter
TCPDump Echo Reply

TShark

TShark is WireShark’s CLI counterpart, supporting the same options as WireShark. In our case the TShark filter format is similar to TCPDump, with exception to the verbose setting. Again, we’ll filter ICMP ECHO (Request/Reply) traffic marked with a ToS/DSCP of 184/46.

TShark ICMP Echo Request Filter
Linux TShark Echo Request

Notice TShark does not display the ToS byte as shown with TCPDump. We can customize this but for now, stop TShark (Ctrl+C) and change protocol filer to look for ICMP ECHO Replies (Type 0). Per the filter matches below, ToS/DSCP of 184/46 is being preserved.

TShark ICMP Echo Reply Filter
Linux TShark Echo Reply

If you’re interested in getting full packet details via CLI, use the -V switch shown below. While the verbose option displays a lot of great detail, the output is much more significant!

     tshark -i eth0 -V ip[1]==184 and icmp[0]==0

TShark Packet Details
Linux TShark Verbose

Adding DSCP column to TShark/WireShark

Unlike TCPDump, TShark/WireShark displays the ToS byte as a DSCP value. To display DSCP without verbose enabled, simply add a new column to display IP DSCP values (first 6-bits of the ToS byte). To add a column, open WireShark and select Edit > Preferences… > User Interface > Columns. Next select Add, enter Field name for column name and select Field type: IP DSCP Value. In the example below, the new DSCP column has been moved between Destination and Protocol columns.

Linux TShark New Colunm

Now, if we open TShark and run the same filters the DSCP value is displayed without verbose enabled.

Linux TShark New Colunm Example

WireShark

WireShark is a protocol analyzer that comes with a GUI, allowing you to analyze network traffic graphically. As with previous examples, we’ll filter ICMP ECHO (Request/Reply) traffic marked with a ToS/DSCP 184/46. Note, the filter syntax is different from TCPDump, with the ip.dsfield.dscp filter matching traffic based on DSCP values (46 in this case) and the icmp.type filter matching ICMP traffic. The following examples display DSCP values, within the packet capture and now in it’s own column.

TShark ICMP Echo Request Filter
Linux WireShark Echo Request

TShark ICMP Echo Reply Filter
Linux WireShark Echo Reply

Posted in Linux, VoIP | Tagged , | Leave a comment

Bandwidth Testing with Iperf

When troubleshooting network related issues and suspect the network at fault it can be challenging at times to finding supporting evidence. One great tool for on-the-spot testing of network bandwidth is Iperf. It’s a simple, yet powerful tool that can provide visibility on network performance metrics such as bandwidth, latency (delay), jitter, and packet loss. While I prefer using the program via the command prompt be aware there is a GUI version called Jperf too.

Today we’ll be using Iperf to measure network performance in terms of bandwidth, but before we get started its worth noting the unit of measurement for file size and transfer rate are different. Files are typically stated in bytes, while transfer speeds are expressed in bits!

When testing bandwidth performance with Iperf, what we’re actually testing is maximum TCP bandwidth at the transport layer (L4). Lots of applications use TCP as the transport protocol, include HTTP, SMTP, FTP, etc. Unlike UDP (another commonly used L4 protocol) TCP is a reliable, connection-oriented protocol with built-in mechanisms for connection established, acknowledgement, and termination. This connection management is why TCP is used (and not UDP) to test bandwidth. If UDP was being used, any lost packet(s) would go missing from the traffic flow and not be resent. Simply put, TCP has a means to detect and retransmit any lost segments.

The following examples use a combination of Microsoft Windows 7 (64-bit) and XP (32-bit). Also, the default TCP window size is used but keep in mind you can adjust window size and effect bandwidth results. To use the program, open the command prompt and navigate to the directory where Iperf.exe is located (Start>Run and type cmd, then press enter). Once in the command prompt you can change directories via the cd command. As you can see I am using Iperf version 1.7.0.

null

Before we get started I want to quickly discuss Iperf’s architecture. It uses a client/server model, where traffic is initiated from the client and traverses the network (LAN and/or WAN) to the server. So when testing bandwidth in both directions we’ll need to run the test twice, once in each direction. Also, by default the server will listen for the client on TCP port 5001 but like most options this can be changed. Note the ability to change TCP ports in Iperf opens avenues to use it for testing open TCP ports. Thus if a firewall or filtering device is blocking port(s) on a segment of your network you can use Iperf to verify this.

To get started we’ll setup the server side first, entering iperf -s -i 1. The -s command designates this workstation as the “server” and the -i 1 command sets the console output interval to one second.

null

Now to start a test, go onto the client and enter iperf -c -i 1. The -c command designates this workstation as the “client” and the IP address of the server. By default, the test runs for 10 secs but can be changed with the -t option. As you can see below a test was run (from client to server) displaying the transfer rate (MB) and bandwidth (Mb) performance every second.

null

Now if we look back at the server you’ll also see similar output as the client. The test output shows Iperf was able to transfer 110MBytes of data at a rate of 92.3Mbits, from client to server.

null

Now to run the test in the opposite direction simply reverse the commands on the client and server. To stop Iperf on the server or client side enter Ctrl+C.

A key point to remender when testing bandwith with Iperf is that it consumes all bandwidth available between clinet/server via TCP, regardless of LAN, WAN, or VPN connection. Thus if traversing a WAN link you can quickly saturate the link, potentially effecting user/application experience!

Below is another Iperf test in the same environment as above but this time the client is wirelessly connected to the LAN. As you can see below, the wireless media has a big impact on bandwidth performace.

null

One last example is of a test performed over a VPN connection. As you might imagine, the bandwidth performace is considerably slower which is attributable to factors such as encryption and Internet speed.

null

Also, you can use a protocol analyzer such as WireShark on the client/server to verify various aspects of the test, such as L4 protocol, Src/Dst ports, packet size, etc.

null

As you can see, Iperf is a simple to use tool that can quickly provide bandwidth performance metrics over a given link. Instead of guessing or suspecting network performance is the issue, with Iperf you can now know!

Lastly, the examples above use the basic options but keep in mind you can change or add numerous options to your testing. Below is the output of the help command to show you the various things you can change.

null

Posted in Bandwidth | 11 Comments

Using Windows to verify DSCP

In today’s converged networks Voice over IP (or VoIP) is a way to provide voice services over packet switched networks, replacing traditional circuit switched networks. While voice communications (as in audio) is a real-time application IP networks have inherent latency and operate in a “best efforts” model. As a result, configuring end-to-end Quality of Service (or QoS) is not only essential but even when done 100% correctly you can still have poor results.

Whether installing a new VoIP system or changing manufactures you should ALWAYS test the network(s) where voice traffic will traverse prior to deployment. Troubleshooting QoS related issues is a complicated task and REQUIRES a solid understanding of many variables. Most commonly you’ll read about latency, jitter, packet loss and their acceptable thresholds. While true there are many other variables to consider so this is just a starting point for testing QoS.

Today, DSCP or Differentiated Services Code Point is a popular model used to classify and prioritize traffic in the network.  Typically the application (in this case an IP phone or voice switch) will mark the DSCP value in the IP header while QoS on the routers/switches will give preferential treatment to packets based on the DSCP value(s) when congestion occurs.  Please be aware there are many ways to implement QoS such as using ToS, IP Precedence, and CoS values and/or incorporating techniques such as ACLs and VLANs to classify traffic.

The industry standard is to use a DSCP value of 46 (aka Expedited Forwarding or EF) to mark/classify real-time applications such as voice.  This test will verify the DSCP value is being preserved between endpoints.  IF the DSCP value is being stripped (typically set to 0) or re-marked to a different value then the traffic will not be classified properly.  This is a simple but effective method of verifying DSCP integrity using readably available software.  For this test you’ll need an OS and NIC that support marking DSCP/ToS bits on the local and remote ends, use of a protocol analyzer, and the trusty ping command. I’ll be using Microsoft Windows XP for the OS and Wireshark as the protocol analyzer.  For XP you’ll need to make a registry change to allow this marking but is easy to do .You can visit http://support.microsoft.com/kb/248611 for more information on this.

To start open the command prompt and type “ping /?

In the output you will notice an option “-v <value>” to mark a ping packet’s ToS (Type of Service) byte.  While this is not the aforementioned DSCP value, the ToS byte (or 8-bits) encompasses DSCP as DSCP only uses the FIRST 6-bits of the ToS byte and ignores bits 7 and 8.  Thankfully with some simple binary math you can translate a ToS value into a DSCP value and via versa.  For this example, we’ll be using a ToS value of 184 (10111000 in binary), which translates into a DSCP value of 46 (101110 in binary).

To generate ping traffic with DSCP of 46 set the ping -v option to 184 as show below.  I also like to use the “-t” option which runs the ping command continuously.  REMEMBER whatever you ping it too must support marking its ToS/DSCP bits.

ping –v 184 <ip-address> -t

Example in the Windows XP command prompt

Now open Wireshark and select the network interface that is generating the ping traffic.

For this test I prefer to filter/limit Wireshark output to ICMP traffic (ping is a type of ICMP traffic) so I only see the traffic I’m interesting in. Also, in Wireshark I recommend creating a column displaying DSCP values. As you can see below the ping packets are both leaving and returning with a DSCP of 46.

If you don’t setup a column for DSCP values then you will need to examine the IP header in each packet to verify the DSCP value. As you can see below the bits in the DSCP field is 101110 which translates into 46… or 2e if your into reading hex.

If you receive the same output in Wireshark as above, then you have just SUCCESSFULLY verified DSCP is being preserved between endpoints. It is HIGHLY recommended to test all paths in which VoIP will traverse the network. Also, it’s a best practice to separate voice and data traffic onto separate VLANs and IP Networks. As such, the endpoints for this test reside on the voice network. Whether you segment voice and data or not the main point is to test the network where voice traffic will travel.

Below is an example where DSCP is being stripped, leaving as a 46 and returning with a value of 0. There could be many reasons for this such as the carrier stripping the value or misconfiguration on a routers/switch. Keep in mind that DSCP stripping might only be occurring in ONE direction or BOTH directions over a particular link.

In the example above, there’s a good chance the remote endpoint (10.0.0.96) is not seeing a DSCP value of 46. This means somewhere between your computer and remote end the DSCP is being stripped. If this is the case, I recommend stopping the test and trying it in the opposite direction to see if you get the same results. Once establishing which direction(s) the stripping occurs move the endpoints closer and closer together over the IP network and retest along the way until you determine where the DSCP value is being stripped and in which direction.

While this test does not perform any type of QoS testing per say, it DOES verify DSCP integrity and should be performed BEFORE using any QoS test tool. Remember if DSCP is being stripped or remarked to an unexpected value then QoS will effectively be meaningless and voice traffic will most likely get “best effort” treatment.

Below are some examples of real voice traffic, captured via port mirroring or SPAN in the Cisco world.

Example of audio traffic’s DSCP being stripped both ways (using G.711u codec)

Example of audio traffic’s DSCP being preserved both ways (using G.711u codec)

Posted in VoIP | Tagged , | 7 Comments