Question about FEX's Cisco Fabric Extender

[Dieselboy]

In FEX's do they allow switching between different ports on the FEX? As I understand, they don't work that way and they forward everything to the Nexus switch that controls them.
I had a quick Google and the FEXs do have ASICs but that doesn't mean they can switch between ports locally.

Looking around some more, and I found this:

The Fabric Extender Forwarding
A FEX or a Nexus 2000 operate as a remote linecard, but does not support local switching, all forwarding is performed on the parent switch.

So with that, if I have a FEX and all ports of the FEX are in the same VLAN. And there are two hosts plugged in to the FEX with a single NIC each and they communicate - all packets are switched through the parent nexus switch correct?

[wintermute000]

yep everything hairpins.

This is why there are so many bleedin options for how to uplink the thing and how to load balance or pin or whatever.

[Dieselboy]

 

Thanks for confirming.

I ask because I was advised to connect our new SAN directly into the FEX. But when I thought about it this came to mind, hence this post.
Technically connectivity will work but if devices in the FEX need access to the SAN then it will hairpin up to the Nexus and back down to the FEX again. Doesn't seem like the best place for a SAN to be connected but I suppose you can argue, with multiple 10GB uplink ports would you notice any difference.

[wintermute000]

Yah I'd stick the SAN into the chassis directly if possible.

It depends what you're talking about - end host IP storage access you should be fine, but iSCSI/FCoE I would stay the heck away.

[icecream-guy]

I think the FEX 2300 series has that port to port switching capability.

if you are running the FEX 2200 series, they have really small buffering capability as I've discovered recently, congestion will cause tail drop and packet loss.  it's best to keep your "good" stuff directly on the 5K and spread across multiple ASIC's

[wintermute000]

I did not know that (i.e. new silicon can actually switch normally). ooh 32Mb buffers.

Can you find a formal reference? I can only find a forum discussion saying its being enabled in a software release. I can't find any references on the datasheet dated august 24th.

[icecream-guy]

I did not know that (i.e. new silicon can actually switch normally). ooh 32Mb buffers.

Can you find a formal reference? I can only find a forum discussion saying its being enabled in a software release. I can't find any references on the datasheet dated august 24th.

no direct reference, but this is what I got from TAC

Code Select Expand


Technical information about Fex 2232:

A 2232 has 4 Switching Subsystems (SS) for the Network to host direction. Each SS connects to all 8 uplinks (NIFs) and 8 host facing ports (HIFs).

One SS has a total of 1280KB of hardware buffer space, out of the 1280KB, only 320KB are enabled per NIF. Two uplinks = 640KB of buffer space, so this is the idea, by adding additional NIFs we gain more buffer. There is a maximum though, and it is reached when the 4th uplinks has been added, for a total of 1280KB of hardware buffer, which as you can see is pretty low.

Pause Concept

• Tx Pause on HIF

The NIF is congested. Example would be two hosts sending a burst of traffic which hashes to the same NIF

• Tx Pause on NIF

A "No-drop" class became congested on a HIF. Example would be too much FCoE traffic on a HIF.


Additional information:

We know that Fex switches are by design oversubscribed devices. When it comes to drops at Fex level, there are a couple of recommendations that would always be the same according to Business Unit, since this is a pretty well documented issue in our database already. Here we are pretty much talking about hardware limitations, and the choices are just a few, if they don't work, moving Hosts to another device like the parent switch itself would be the only fix.



[Dieselboy]

I think the FEX 2300 series has that port to port switching capability.

if you are running the FEX 2200 series, they have really small buffering capability as I've discovered recently, congestion will cause tail drop and packet loss.  it's best to keep your "good" stuff directly on the 5K and spread across multiple ASIC's

I plan on connecting straight into the Fabric Interconnect - this runs NXOS and has almost 1Tb of switching throughput capacity. Not sure on the buffers there but I expect they're beefy to cope with the throughput

The 2200 FEX's are inside the Cisco UCS 5108 chassis. The FEX connect back to the FI's.

[wintermute000]

disclaimer: we're talking about 5/7ks. I know FIs are very similar under the hood, but, I can't tell you with any certainty whether the UCS stuff is any different from the NExus stuff.

[Dieselboy]

Yea I'm not sure either, hence mentioning it for you guys to tell me if I'm wrong or not
 

[NetworkGroover]

In FEX's do they allow switching between different ports on the FEX? As I understand, they don't work that way and they forward everything to the Nexus switch that controls them.
I had a quick Google and the FEXs do have ASICs but that doesn't mean they can switch between ports locally.

Looking around some more, and I found this:

The Fabric Extender Forwarding
A FEX or a Nexus 2000 operate as a remote linecard, but does not support local switching, all forwarding is performed on the parent switch.

So with that, if I have a FEX and all ports of the FEX are in the same VLAN. And there are two hosts plugged in to the FEX with a single NIC each and they communicate - all packets are switched through the parent nexus switch correct?

Yes - that's absolutely correct.  Something we make sure customers are aware of when we suggest alternatives.