Jumbo Frames for storage <->

Started by Dieselboy, April 22, 2016, 07:31:19 AM

Previous topic - Next topic
Print
Go Down Pages1 2
User actions

Dieselboy

April 22, 2016, 07:31:19 AM
When I originally set our network up in 2013 I enabled Jumbo frames on the Nexus core and the Fabric Interconnects where the storage and UCS chassis are for 9000byte packets for switching between the SANs and the virtualisation system.
The storage guy and server guy didn't enable jumbo frames so although the underlying switching network supports it, it's not in use. We don't need jumbo frames, at the moment. IOPs on our netapp sans only goes up to 4k when we're running the nightly backups, otherwise it's in the hundreds on both / either HA controllers during the day. I'm graphing CPU of the netapps and I occasionally see spikes up to 30% and may be once a day in the 40% range.

I have a 10GB switching network with Fabric Interconnects. If I do enable jumbo frames, could I expect to see less CPU use since there is less time spent reading / writing header info? Without jumbo frames I expect for 9000 bytes, 6x the header info.

I would only see a performance increase if the current throughput is being limited by time spent writing headers etc right? IE if the disks cant write data any faster then I'm not going to see a write speed improvement with jumbo frames. The switches would be cut-through since they are 10GB so the network throughput wouldn't improve with jumbo frames. Is my thinking correct?

Is anyone using jumbo frames? Are you using them on 10GB links? I would have preferred to have them enabled at my place from day 0, that way there's no risk of enabling it later on and breaking everything :)

The reason I was thinking about this is that I've just moved a bunch of network share drives off legacy storage and onto the SAN and I'm seeing single file write speeds at the OS level of around 100MB/s using an iSCSI connection. Monitoring the storage NIC of that VM I'm seeing occasionally 1gbit/s and just over, for the main reason that it couldn't pull the data from the legacy storage any quicker and it just got me thinking if it was possible to optimise the SAN. Not that it needs it though.

deanwebb

#1
April 22, 2016, 07:40:29 AM
"jumbo frames"

I just love saying jumbo frames. We use them in traffic within the data center, but not outside. After that, I can't say much except that whenever something breaks with them, I get to say that there isn't a firewall in the path... so...

:notthefirewall:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#2
April 22, 2016, 08:14:58 AM Last Edit: April 22, 2016, 08:18:16 AM by Dieselboy
I just found this evening that jumbo frames at layer 3 is called a jumbogram. I don't want jumbograms. Strippergrams, maybe... but not jumbograms.
:problem?:

deanwebb

#3
April 22, 2016, 08:16:39 AM
What's the header information on a strippergram?



OK, sorry about that... I hate when I have to warn myself for an off-color post... :mrgreen:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#4
April 22, 2016, 08:22:13 AM
haha I was thinking that as I was reading :)

PS Is your datacentre using 10GB links?
I'm thinking that jumbo frames with cut-through switching is going to be a different experience than jumbo frames with store and forward. I'd like to discuss that :)
Store and forward would use up more buffers. This is the problem with 3750 switches as SAN switches for example. By default ie if you don't tweak the buffers you will overload the buffers per port in some situations and cause big issues. I've still got half a draft email from 2012 where I went to write to a very senior colleague about what I'd found RE buffer configuration and how it could help with an ongoing issue. But the email got huge and I didn't want to pee him off so it never got sent :) It didn't matter anyway as the SAN provider eventually provided the server team with similar info.

deanwebb

#5
April 22, 2016, 08:43:02 AM
10GB links for the boxes that need them. Everyone else can live with a gig link.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#6
April 22, 2016, 10:21:22 AM
Before I joined my company, the developers were pushing for 10GB to the desktop

:zomgwtfbbq: :lol:

icecream-guy

#7
April 22, 2016, 11:04:16 AM
Quote from: deanwebb on April 22, 2016, 08:43:02 AM
10GB links for the boxes that need them. Everyone else can live with a gig link.

with all the voice, video, steaming, and other application bandwidth hogs, 10GB to the desktop will be the norm soon
:professorcat:

My Moral Fibers have been cut.

NetworkGroover

#8
April 22, 2016, 11:49:20 AM
Quote from: ristau5741 on April 22, 2016, 11:04:16 AM
Quote from: deanwebb on April 22, 2016, 08:43:02 AM
10GB links for the boxes that need them. Everyone else can live with a gig link.

with all the voice, video, steaming, and other application bandwidth hogs, 10GB to the desktop will be the norm soon

Gotta get your HD multi-angle pr0n on.
Engineer by day, DJ by night, family first always

Dieselboy

#9
April 22, 2016, 01:47:38 PM
When it's affordable then definitely. But when we're barely pushing 1GB links, 10GB is not worth the cost for us :)

icecream-guy

#10
April 22, 2016, 04:02:28 PM
Quote from: Dieselboy on April 22, 2016, 01:47:38 PM
When it's affordable then definitely. But when we're barely pushing 1GB links, 10GB is not worth the cost for us :)
shall I repeat....



Gotta get your HD multi-angle pr0n on.
:professorcat:

My Moral Fibers have been cut.

deanwebb

#11
April 22, 2016, 04:23:16 PM
Quote from: ristau5741 on April 22, 2016, 04:02:28 PM
Quote from: Dieselboy on April 22, 2016, 01:47:38 PM
When it's affordable then definitely. But when we're barely pushing 1GB links, 10GB is not worth the cost for us :)
shall I repeat....



Gotta get your HD multi-angle pr0n on.



Lol, I might have to make that a permanent addition to the vast array of smilies...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

NetworkGroover

#12
April 22, 2016, 05:00:51 PM
Lol... I love this forum.... even if all I do half the time is annoy the crap out of Dean....
Engineer by day, DJ by night, family first always

deanwebb

#13
April 22, 2016, 05:23:27 PM
Quote from: AspiringNetworker on April 22, 2016, 05:00:51 PM
Lol... I love this forum.... even if all I do half the time is annoy the crap out of Dean....

I don't get annoyed here. I laugh. When I want to get annoyed, I go involve myself in the endless struggle between historians and gamers at the Europa Universalis IV forums... lots more people there that have no clue how to "roll with it."

Side note: networkers, in general, are not a highly-strung crowd. I don't have to explain jokes to them like I have to with the programmers...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

mlan

#14
April 22, 2016, 06:38:34 PM
I recently deployed a 10GbE iSCSI fabric on Nexus 3500's and enabled jumbo frames to make all the compute/storage teams and vendors happy.  Here is some reading material, along with some benchmarks in the two older articles:

https://www.reddit.com/r/networking/comments/3nvvrw/what_advantage_does_enabling_jumbo_frames_provide/
https://vstorage.wordpress.com/2013/12/09/jumbo-frames-performance-with-iscsi/
http://longwhiteclouds.com/2013/09/10/the-great-jumbo-frames-debate/
Print
Go Up Pages1 2
User actions