Tip of the Month

Network Analysis Tip of the Month – June 2006

When Is a Retransmission Not a Retransmission?

By Jeff Trawick, WildPackets Professional Services

TCP retransmissions are pesky! Although all “normal” networks will experience a few of these, more than one retransmission per megabyte of data is usually, in technical terms, a really bad thing. TCP retransmissions can have a variety of causes, but the most frequent culprits are physical layer issues, such as cable, connector, and port faults, or electromagnetic interference of some sort. Anything that could corrupt a packet and cause it to be discarded can cause a TCP retransmission. The packet is never received at its intended destination, the recipient never sends a TCP ACK, so whoever transmitted the packet is obligated to try again.

The Expert in the Peek analyzer readily recognizes and flags retransmissions when it sees a packet with a previously used TCP sequence number in a specific conversation. The Expert will also report that it sees “Too Many Retransmissions” when the percentage of retransmitted packets exceeds 10% of the total packets. When these events are reported by the Expert, it is usually time to pull out the network firefighting gear and try to follow the smoke to the blaze. But are there cases in which the analyzer may have a false positive for TCP retransmissions? The answer is: YES! Here’s one example.

Recently, while performing an onsite analysis for a WildPackets client, we noticed behavior that we initially thought was really scary – tons of TCP retransmissions over fairly short time spans, affecting multiple conversations. When we looked at these packets, something unexpected appeared. A packet with the same sequence number was being repeatedly retransmitted AND it was being acknowledged each time. Even more puzzling was the fact that the ACK numbers were exactly the same as the previous sequence numbers (they should increment by at least 1, even when no data is being transferred). All of these packets were minimum-sized (64 byte) packets. Take a look at the Visual Expert Packet Visualizer for this conversation.

Click on thumbnail for larger view

Notice that all of the sequence and ACK numbers are exactly the same – seemingly very strange! But also notice the frequency of the packets in the Delta Time column. Beginning with packet 738, a pair of packets from the client and server occur about approximately every 5 minutes, a pattern that can’t be easily ignored. Some of the conversations were doing this about every 2 seconds! So what’s causing this activity?

After a little research, we discovered that this was a normal TCP keep-alive process – in this case, for a SQL server process. Once identified, this information allowed us to ignore the bulk of the retransmission events in the Expert, and focus on other events that led us more quickly to the real problem on the network. So keep in mind that when you hear hoof beats, it ain’t always horses! When the Expert reports something bad, be sure to perform a thorough investigation into the matter before assuming the worst!