Transfer speed on N5K

[wintermute000]

When even SAN guys say stuff jumbo... https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/

[killabee]

I haven't found a clearer explaination of this, but gig links and above NEED auto-neg in order to achieve that level of throughput (or close to it).  Augo-neg in those cases do more than just negotiate the speed/duplex.  It determines the master/slave relationship between the connected endpoints, determines the flowcontrol, etc, all of which play a part in the throughput.  That's probably why you're seeing poorer performance at full-duplex over auto.

http://h10025.www1.hp.com/ewfrf/wc/document?docname=c01148835&cc=lb&dlc=en&lc=fr

Are those throughput numbers with full-duplex, or auto?

As for the jumbo vs no jumbo.....we network engineers know what's up.  It's the server team, storage team, and their vendors that pressure us to enable it. 

[ZiPPy]

I've always gone by best practice for SAN infrastructures, and that was to enable jumbo frames.  This is with the notion that you keep the traffic contained, no default gateways.  The benefits certainly shine in file transfers, respectively. 

I'm not saying it's a must, or that one way is better than the other.  It's more so configuring and tuning the network to meet a specific need in your network, in your data center.  Enabling jumbo in one network, might not reap the benefits in another. 

I stumbled across this paper awhile ago, that really gets down to the nitty gritty of jumbo.  http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=2770&context=cstech

[sgtcasey]

During WAN bandwidth testing on new links I usually use jperf/iperf with UDP packets and set the amount of traffic way past what the link speed we bought is.  Then I just check the other end interface to see how much is actually getting through.  This might be a better way to test the true throughput of the links on the Nexus stuff.  Just an idea.

I use N5K's with FEX switches in my work environment and I've never seen issues that are described by the OP.  It does make me wonder, though...

[burnyd]

If iperf is working fine and you are getting the speed you are expecting and this is a tcp based application a lot of times before I even troubleshoot cifs/smb I always doublecheck to make sure tcp off load is turned off at the OS level. 

Otherwise, it can be a large amount of things.  If you are handing off to fex's and you still have everything default hte default queue buffer is one large buffer spread across every port.  I would disable that for safe measures depending on the model of FEX you will only see for example a max of 2.25gbps out of a port which would explain your 220mb/s I believe you said prior.  If you can also please take a look at the pause frames under the interface and tell me if it is a large number?

[fsck]

During WAN bandwidth testing on new links I usually use jperf/iperf with UDP packets and set the amount of traffic way past what the link speed we bought is.  Then I just check the other end interface to see how much is actually getting through.  This might be a better way to test the true throughput of the links on the Nexus stuff.  Just an idea.

I use N5K's with FEX switches in my work environment and I've never seen issues that are described by the OP.  It does make me wonder, though...

May I please know what command you ran for your iPerf? I know the command can give you false results if you write it wrong.  So I want to be sure that's correct.  This is what I'm running iperf -c 172.1.1.10 -p 5001 -t 15 -i 1 -f m -w 50MB

[fsck]

If iperf is working fine and you are getting the speed you are expecting and this is a tcp based application a lot of times before I even troubleshoot cifs/smb I always doublecheck to make sure tcp off load is turned off at the OS level. 

Otherwise, it can be a large amount of things.  If you are handing off to fex's and you still have everything default hte default queue buffer is one large buffer spread across every port.  I would disable that for safe measures depending on the model of FEX you will only see for example a max of 2.25gbps out of a port which would explain your 220mb/s I believe you said prior.  If you can also please take a look at the pause frames under the interface and tell me if it is a large number?

We aren't using any fex's. I'm seing 0's for the pause frames.  I made sure i enabled the flowcontrol.  Not seeing anything for the pause frames is a good thing right?

[sgtcasey]

May I please know what command you ran for your iPerf? I know the command can give you false results if you write it wrong.  So I want to be sure that's correct.  This is what I'm running iperf -c 172.1.1.10 -p 5001 -t 15 -i 1 -f m -w 50MB

Sure.  I'll set up a laptop connected to the remote router I want to test WAN bandwidth over.  On that laptop I start up iperf -s (for TCP) or iperf -s -u (for UDP).  Then on my work-issued machine I'll go to the other end of that WAN link and depending on what I want to use I'll run the following.

TCP - iperf.exe -c <iperf server IP> -P 1 -i 1 -p 5001 -f m -n 1000000000
UDP - iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 200.0M -n 1000000000 -T 1

I use TCP to see what kind of performance a normal TCP session will get over a link.  For UDP I tell it to just spam tons of traffic through the WAN link.  I then check the other side interface stats to see what is actually making it through.  You do need to change how your iperf server is set up depending on if you want to use UDP or TCP.

[fsck]

So I ran the command as you have shown sgtcasey and I'm getting the following results

TCP - iperf.exe -c <iperf server IP> -P 1 -i 1 -p 5001 -f m -n 1000000000
Transfer               Bandwidth
144 MBytes       1208 Mbits/sec
180 MBytes       1514 Mbits/sec
148 MBytes       1237 Mbits/sec
176 MBytes       1481 Mbits/sec
177 MBytes       1486 Mbits/sec
954 MBytes       1394 Mbits/sec

UDP - iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 200.0M -n 1000000000 -T 1
Transfer               Bandwidth
24.1 MBytes       202Mbits/sec
24.1 MBytes       202Mbits/sec
24.5 Mbytes       206Mbits/sec
24.1 MBytes       202Mbits/sec
24.1 MBytes       202Mbits/sec
24.1 MBytes       202Mbits/sec
24.1 MBytes       202Mbits/sec

Could somebody else post up there results?  I'd be curious to compare.  But based on what burnyd said, there could be all kinds of reasons why the performance is lacking.

If you take a gigabit Cisco Catalyst switch, you can pretty much power it up and it's ready to go.  No port configuration necessary, and you'll get gigabit speeds.  Is that not the same for the Nexus 5Ks?  Do I need to specify certain parameters on it?  So far all I've configured on it, is the policy for jumbo frames.  I'm going to look around and see if I missed a configuration somewhere that might limit the speeds.

[fsck]

I still wonder what I'm doing wrong.  Was anybody able to run the test too?  I know it will vary because we don't have the same NICs or possibly different cabling, but it should be around the same area.  I'm pretty sure higher than what I have now.  How does one troubleshoot an issue like this?  I used iPerf to prove we have a problem, so now what do i turn to?  I know in class they say use OSI model.  Not sure how to go about that at this point.

[javentre]

post a 'show interface' of the involved ports

[killabee]

Are those throughput numbers with full-duplex, or auto?

[fsck]

post a 'show interface' of the involved ports

Code Select Expand

Ethernet1/1 is up
Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 0005.73f0.b828 (bia 0005.73f0.b828)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is access
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is on, output flow-control is on
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 4d05h
  Last clearing of "show interface" counters never
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 752 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 352 bps, 0 pps
  RX
    115894665 unicast packets  10167 multicast packets  16803 broadcast packets
    115921635 input packets  233682617296 bytes
    8635860 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    79116716 unicast packets  1108990 multicast packets  465984 broadcast packet
s
    80691690 output packets  102570428844 bytes
    4899 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  7 interface resets


Code Select Expand

Ethernet1/13 is up
Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 0005.73f0.b834 (bia 0005.73f0.b834)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is access
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is on, output flow-control is on
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 1d06h
  Last clearing of "show interface" counters never
  30 seconds input rate 464 bits/sec, 0 packets/sec
  30 seconds output rate 328 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 32 bps, 0 pps; output rate 200 bps, 0 pps
  RX
    8324552 unicast packets  14437 multicast packets  445021 broadcast packets
    8784010 input packets  9361511697 bytes
    0 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    25630187 unicast packets  529285 multicast packets  18281 broadcast packets
    26177753 output packets  100467975215 bytes
    8633316 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  21 interface resets


[fsck]

Are those throughput numbers with full-duplex, or auto?

Currently with full-duplex.  I tested with auto but I was getting the same results.  Wouldn't you want it set to full-duplex though?  Why and when do you use auto?  I always though you set it to full-duplex to be safe.

[javentre]

Hard coding hasn't been a recommended practice for about 15 years.   Always use auto/auto where possible, especially at speeds >= 1gbps.