Transfer speed on N5K

[fsck]

I'm noticing that on our gigabit network we can hold about a 84MB/s transfer rate, which is about the max of a 1GB connection being that 125MB/s is the theoretical maximum.  My question is on the N5Ks I see it hit about 921MB/s (max theoretical 1250MB/s) and it starts to drop rapidly and hold around 140MB/s.  Transferring a file about 125GB in size. We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

Something odd I noticed, when I set the Intel NIC settings to full-duplex, it no longer peaks at 921MB/s maybe up to about 260MB/s and comes and holds around 133MB/s.  I'd like to learn what exactly is going on, and why these traits are showing.  And of course I'd like to fully take advantage of the 10GB network. The 140MB/s isn't that exciting, I was expecting it to take off.

[Seittit]

We've noticed ridiculously poor performance on our FEX modules and TAC found it related to its internal hardware queues. With our situation, Nexus automatically allocates a large amount of hardware queues for FCoE (even on non-FCoE capable devices such as our 2148 FEX modules). This led to large amount of drops on the transmit direction.

We found that a home netgear switch continually outperformed our Nexus switches, try delivering that nugget of news to management.

I would recommend opening a TAC to ensure if a similar issue isn't affecting your 5Ks.

[icecream-guy]

We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

shouldn't the server NIC MTU be the same size as the jumbo frame MTU ?

[javentre]

It depends on what you want to accomplish.  As long as the switch is set higher than the end hosts, you shouldn't drop frames.

[javentre]

I'm noticing that on our gigabit network we can hold about a 84MB/s transfer rate, which is about the max of a 1GB connection being that 125MB/s is the theoretical maximum.  My question is on the N5Ks I see it hit about 921MB/s (max theoretical 1250MB/s) and it starts to drop rapidly and hold around 140MB/s.  Transferring a file about 125GB in size. We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

What tools are you using to test throughput?

[fsck]

We've noticed ridiculously poor performance on our FEX modules and TAC found it related to its internal hardware queues. With our situation, Nexus automatically allocates a large amount of hardware queues for FCoE (even on non-FCoE capable devices such as our 2148 FEX modules). This led to large amount of drops on the transmit direction.

We found that a home netgear switch continually outperformed our Nexus switches, try delivering that nugget of news to management.

I would recommend opening a TAC to ensure if a similar issue isn't affecting your 5Ks.

Won't this issue be a little different?  Your FEXs are like your remote line cards.  So we are talking an added piece of connectivity, one that I'm not dealing with if that makes sense.  We are simply treating our N5K right now as a core switch, and trying to maximize the throughput.  I only wish we had TAC.

[fsck]

We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

shouldn't the server NIC MTU be the same size as the jumbo frame MTU ?

My mistake.  I'm so used to writing 1500 MTU for my labs.  I changed the NIC to 9014 bytes.

[fsck]

I'm noticing that on our gigabit network we can hold about a 84MB/s transfer rate, which is about the max of a 1GB connection being that 125MB/s is the theoretical maximum.  My question is on the N5Ks I see it hit about 921MB/s (max theoretical 1250MB/s) and it starts to drop rapidly and hold around 140MB/s.  Transferring a file about 125GB in size. We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

What tools are you using to test throughput?

tbh javentre, this was a simple Windows copy which I know isn't a true test, kind of.  I'm looking at setting up IOMeter and see what that says.  Any tools you think are best?

[ZiPPy]

I'm noticing that on our gigabit network we can hold about a 84MB/s transfer rate, which is about the max of a 1GB connection being that 125MB/s is the theoretical maximum.  My question is on the N5Ks I see it hit about 921MB/s (max theoretical 1250MB/s) and it starts to drop rapidly and hold around 140MB/s.  Transferring a file about 125GB in size. We are using jumbo frames and I've verified the NIC on the server is set for 1500 MTU.

What tools are you using to test throughput?

tbh javentre, this was a simple Windows copy which I know isn't a true test, kind of.  I'm looking at setting up IOMeter and see what that says.  Any tools you think are best?

Use iPerf to test throughput and use something like Steelhead Packet Analyzer.  Do a Wireshark capture, then run results through Steelhead Packet Analyzer. which can show you problems with TCP and latency. iPerf won't show you problems with TCP and latency, just throughput so the combination could be beneficial. Definitely start with iPerf first.

[that1guy15]

There is so many different things this could be. I am assuming this test is between two ports on the N5K right? What model?

Why jumbo frames? Why?

First make sure the switch is actually getting overloaded. Do you see buffer misses? Do you see queuing and drops on the interfaces?

Next double check your source and destination hardware can handle this type of network and processing load. I have never been able to take a standard desktop or laptop and truly test 1Gbps.

A standard file transfer from windows to windows is going to add so many variables from the Windows side. Try iperf.

[javentre]

Use iPerf to test throughput

Exactly.  Repeat all of your tests with iPerf, and make sure things are repeatable.

[hizzo3]

We found that a home netgear switch continually outperformed our Nexus switches, try delivering that nugget of news to management.

Hey those GS-108T (v2) are no joke lol.

[fsck]

There is so many different things this could be. I am assuming this test is between two ports on the N5K right? What model?

Why jumbo frames? Why?

First make sure the switch is actually getting overloaded. Do you see buffer misses? Do you see queuing and drops on the interfaces?

Next double check your source and destination hardware can handle this type of network and processing load. I have never been able to take a standard desktop or laptop and truly test 1Gbps.

A standard file transfer from windows to windows is going to add so many variables from the Windows side. Try iperf.

This will be used for an iSCSI SAN network, so that's why we are going with jumbo frames.

iPerf is showing the following ...
7,632Mbit/sec
2990Mbit/sec
2104Mbit/sec
2205Mbit/sec
2102Mbit/sec

So it spikes up to where it should be, it just won't hold.  I have the iperf command as iperf -c 192.168.10.10 -p 5001 -t 15 -i 1 -f m -w 50MB and I tried it with 500MB too.

[fsck]

Just so I'm not mistaken, when you say buffer misses do you mean the 'no buffer' option?  I see 0 across the board for runts, CRC, no buffer, ignored, ect...  So it looks to be clean.  I doubt the N5k is overloaded, as these are the only 2 test servers running the iPerf test.

[hizzo3]

I was having issue with my iSCSI connected to a win2k12 vm at home that was doing similar with jumbo frames... I was seeing a bunch of error packets in wireshark (can't remember the details). The switch wasn't seeing errors, it was something to do with the iSCSI portion. Keep in mind iSCSI isn't a layer 2/3 protocol so I don't know enough about the N5K to tell you if it is processing any higher level protocol errors.

I rolled it back to 1500 MTU and the errors stopped and it stabilized the speed. I didn't have time to trouble shoot it right now since the lab I am doing is fine with non-jumbo.

Did you try a packet capture using wireshark while doing iperf? I'm willing to bet that your having a bunch of unneeded traffic somewhere that is bogging down as the transfers get going.