TCP Windowing Issue

cisco_trooper · March 2012

OK guys I've been fighting an issue for a while now that I can't seem to resolve.

Scenario 1:
I download a file from a CIFS share on a SAN and get blazingly fast speed. Wireshark reveals proper TCP Windowing negotiations and everything works as expected.

Scenario 2:
I upload a file to the same CIFS share on a SAN and get the worst F'ing throughput I've ever seen. TCP Windows aren't negotiating to what I would consider normal and get about 50K/s throughput.

I have tried tuning the TCP parameters on both Windows XP and Windows 7 machines to see if I can get the one way throughput issues resolved but I have had ZERO luck and my brain is about to bleed.

It should be noted that I can reproduce this via LAN access as well as WAN access over IPSec VPN.

Any thoughts anyone? Has anyone else run into this problem?

ptilsen · March 2012

Just to clarify, have you verified the problem is specific to CIFS? I don't see in your list of steps that you've downloaded a file directly to local storage on the server hosting the CIFS share.

Is the SAN storage on a physical machine running Windows, a virtual machine running Windows, or something else? What version of Windows? What SAN protocol? What hardware?

Edit: Also, any noteworthy software on server or client OS? I've seen AVG Link Scanner, for example, cause some strange captures in Wireshark for CIFS and HTTP traffic.

cisco_trooper · March 2012

The SAN storage in question is CIFS advertised by and EMC NS-120. I am going to try shares off Windows File Servers as well as trying something else like FTP, but I'll have to setup an FTP server or something specifically for that. I am accessing the shares from Windows XP and Windows 7, but the initial report of the issue come in from someone using a MacBook Pro, which I don't have available for testing.

I have Kaspersky AV on my workstations and servers, but I won't even get into that whole fiasco at the moment as that is outside my control.

I'll keep this thread updated with details as they become available.

Thanks.

vinbuck · March 2012

I would probably be getting iperf out and setting the TCP settings to match the application. Then you can prove the network is or isn't the problem and either fix it or move on to the OS. Might want to start checking layer 2 and 3 mtu settings along the path also...

SteveO86 · March 2012

vinbuck wrote: »

I would probably be getting iperf out and setting the TCP settings to match the application. Then you can prove the network is or isn't the problem and either fix it or move on to the OS. Might want to start checking layer 2 and 3 mtu settings along the path also...

Yep that is where I would.

Any retransmission or or errors in the Packet capture?

cisco_trooper · March 2012

Sorry for the delay in update. I was able to get a packet capture going at the remote end of the IPSec Tunnel and prove that packets were being dropped. There were a ton of retransmission originating from that end that were never received on my local end resulting in drops in the TCP Window as well as bandwidth wasted on multiple copies of the same segment. Always do your sanity checks folks before you start getting into the workings of the protocol negotiations.

Thanks all.

SteveO86 · March 2012

It's easy to diagnose the problem by looking at the packets, the challenging part is finding why the packets are behaving the way they way there.

(Sadly I've had 3 incidents in the last months forcing me to go through TCPDump output diagnosing some pretty stupid issues so this is fresh in my mind... of course the fun part is when people argue with your findings and the packet capture)

TCP Windowing Issue

Comments