Fabric Interconnect port-channels

[Dieselboy]

With Nexus switches you can set up a VPC peer-link and after that, you can set up VPC port-channels meaning you can have a single host (example) with two separate links (one to each nexus switch) and you can bundle those separate links which connect to separate nexus switches into a single 2GB or single 20GB port-channel.

I also have two Fabric Interconnects which run a version of the NXOS operating system. Can I do something like a VPC... OR can I do a port channel to a single host which connects to both Fabric Interconnects separately?

[that1guy15]

You can do either.

In the past it was recommended to either vPC your FEX and single home your server or dual-home (vPC) your servers and single home your FEX. Everyone I talk to now says single home your FEX and vPC your servers. Ive also seen both but its gets ugly.

If you have single nic servers (yes they are still out there) then it would make more sense to vPC your fex so the server is not lost during a 5K failure. If you have dual nic servers I would single home the fex and vPC the host.

(jumps on soap-box)
Moving forward if I ever have to deploy FEX again (which I hope is never) I will push as hard as possible to single home the FEX and force the server team to setup Port-channels and vPC. Each time Ive deployed in the past the server guys have been able to shift the failure responsibility off on the network and we are required to design around it. They have done this by only buying single NIC servers or insisting they will only uplink to a single switch or will not use LACP. This is why you see all these options with FEX. Dont fall in this trap. Follow best design and stand your ground. Now I also understand how the real world works, so...

[Otanx]

As someone who has deployed FEX in both dual and single attached modes go single. We had a lot of issues with the dual homed FEX setup. In short all the problems came from having two control planes trying to manage a single device. If you forget a step in the upgrade all your FEX are going to reboot at one time.

(joining that1guy15 on the soap-box)
Always dual home servers. Even if you decide to dual home the FEX it is supported to also dual home the servers. Some upgrades require reboots of the FEX, and at that point it does not matter if the FEX is dual homed or not. If you can not dual home the server for some stupid reason (looking at Oracle) then make sure the server people do failure testing to make sure host redundancy works as expected. This should be done anyway, but is even more important if they don't dual home. You don't want your maintenance to be when they find out that a fail over to the secondary server causes database corruption (looking at Oracle again).

-Otanx

[Dieselboy]

Thanks guys and otanx

BTW the only FEX I have is in the server chassis and it's separate, for the chassis.

With regards to Fabric Interconnects (not fex) and vpc, my idea was to VPC a host (server / san / something) to the Fabric Interconnects for resilience and throughput but it seems that the fabric interconnects do not do VPC.

I have this set up:

[n3k] --20gb channel-- [n3k]

||        //         ||          //
[FI-1 vpc1], [FI-1 vpc2]

So to explain because my text diagrams are nowhere near as good as all of yours;

I have a core made up of 2x n3k switches, with a 20GB port-channel linking the two and a vpc peer link set up already (i have vpc channels from the n3k's going to all sorts of places

In addition I have 2x fabric interconnects. FI1 has a vPC channel to EACH nk3 (ie FI1 has 10GB to n3k1 and 10GB to n3k2, making a 20GB vPC1). The second FI has the same vPC set up. This is all good.

The Fabric Interconnect DOES support port channels but doesn't seem to support any vPC type stuff.

So lets say I want to connect in a SAN and make it resilient, I can't do a port channel. To make a 20GB channel, I can't make it resilient.

This is my predicament

[ps. I'm going for resilience, the old SAN had MAX peak bandwidth at 400mbps on the 10GB link so yea..(unless cacti is lieing).

[wintermute000]

Fabric interconnects are not switches at least not in end host mode. Most of the advice given in this thread is off-topic as they're talking about Nexus, not fabric interconnects.
This is a UCS specific topic not regular switching.

They're designed to provide two separate fabrics and vpc is not the point nor doing anything resembling layer two tricks. They're simply there to aggregate your vmnics/vnics ( god I can't stand the UCS/vmware terminology mixup). Providing two independent fabrics is by design (think FC SAN and I mean the storage network not the array!)

If you absolutely have to load balance (why... Esp as you have vpcs northbound), do it at the host layer design level e.g. Split your port group uplink sequences or use load based teaming or SRCID etc.

I was chatting with the guys today re: how the additional complexity of UCS (compared to Dell or HP blades) is basically pointless overhead for most small/ medium enterprise deployments where the hardware stays mostly static. Especially as you're already abstracting with a hypervisor layer, so there goes half the benefits - so what if you can easily replace a blade with a service profile, you're just going to slap vmware on it and its the VMs that matter, if it was a pizzabox you'd just add the host to the cluster and BAM. One guy basically called it an idea that was 10 years too late - it would have been an absolute killer pre-virtualisation.

This discussion is a classic example. If this was HP or Dell we would be dealing with regular switches and everyone would be crystal clear. But now you have this not-switch in the middle you have to learn about and design specifically around....

EDIT sorry got OT again LOL but just triggered due to coincidental discussions today

[that1guy15]

Wow I totally missed that you specifically said FIs and not FEX... Must not have had enough coffee yesterday when I saw this or was just in the mood to bitch about FEX

I agree with wintermute000 on FIs. Basically with FIs you want to uplink as many connections from chassis to FI as you have ports/cables and licenses for. Nothing special on your end the FI handles the fabric for you.

Sorry for the mix-up

[Dieselboy]

This discussion is a classic example. If this was HP or Dell we would be dealing with regular switches and everyone would be crystal clear. But now you have this not-switch in the middle you have to learn about and design specifically around....

EDIT sorry got OT again LOL but just triggered due to coincidental discussions today

Perfect! Excellent helpful response, thanks.

I don't have to load balance I'm just trying to maintain throughput and resilience.

So basically then we treat the FI's somewhat like FEX's to some extent, may be advanced FEX?

If you had a single host which is a VMWare server, say UCS-C or something similar how would it be connected to the topology I described? Because my feeling is that you would port channel to the network for resilience and HA but it seems that is wrong now? You can't port channel if you want to connect to both FI's. And if you connect to only one FI and that FI goes down then you lose your host. So it seems that for this to work properly, you would need at least a single connection to each FI from the host and therefore you cannot port channel and so you cannot obtain 20GB or 2GB bandwidth. The 2GB bandwidth might be a better example for this because a host can easily exceed 1GB compared to 10GB.

In my situation specifically, I have a SAN with two controllers. Each controller has 2x 10GB ports (port 1 and port 2). So I have a total of 4x 10GB ports but at the moment I'm just using port 1 on each controller (the IP fails over from controller 1 port 1 to controller 2 port 1 during a controller failover).
While I'm waiting for additional fibre patch leads to arrive so I can cable up port 2 from each controller, I've connected as follows:
> controller 1, port 1 == FI 1
> controller 2, port 1 == FI 2

I have a single 10GB link active. If FI-1 dies or the cable is pulled, the SAN controller fails over and the IP's move to controller 2 port 1. However the mac address changes requiring a re-arp from the Hypervisors and the network. The time taken for the connectivity to be restored is 4 ICMP drops or around 8 seconds.

When I have the fibre patches I can do the following:
> controller 1, port 1 == FI 1
> controller 2, port 1 == FI 1
> controller 1, port 2 == FI 2
> controller 2, port 2 == FI 2

The above is as-per the SAN documentation. I need to test what happens when FI-1 dies with the above scenario, but with the additional connections there should already be iSCSI through controller 1 port 2 through FI-2 so failover should be seamless. I don't think you can bond / port-channel across SAN controllers though.

But what about the single VMWare condition I mentioned? That's confusing.

[wintermute000]

Again I'm not a UCS guy so perhaps go for specialised advice, but here goes anyway

- You wouldn't be involved with a FI if you had a pizzabox, it would be straight to the ToR... If this is a specialised UCS thing (C series to FI just for mgt or whatever) then there should be clear cut Cisco design guides.
- In a traditional FC there would be two separate fabrics and it would be designed to run like that.
- Now with iSCSI to FIs, I THINK it would be best to use multipath and keep both IP connections open at the same time, but along separate paths. No layer 2 funny business, ever.
- If it was iSCSI to a filer outside of the FIs then no issues you're covered.

https://supportforums.cisco.com/document/133666/ucs-best-practices-and-common-recommendations-configuration-and-management?buffer_share=a0b1d&utm_source=buffer&utm_medium=twitter&utm_campaign=Buffer%253A%252Bvdmchallenge%252Bon%252Btwitter

Storage:

Leverage Boot from SAN to enable a stateless infrastructure
Utilize default pinning (round robin) for server vHBA to uplinks
Leverage Boot Policies to distribute initial boot traffic across backend storage array ports
Use VSAN names and IDs that correlate to upstream fabrics (where applicable)
Leverage redundant SAN fabric switches
Use host-based multipathing for redundancy and load-balancing
Pre-provision SAN resources where applicable to reduce deployment time and enhance stateless infrastructure

[Dieselboy]

Thanks

We will achieve multipath with the additional fibre patches.