Cisco ACI vPC Scenario

[LynK]

I cannot find this problem, nor a solution. But if you have remote leafs in ACI running vPC, and the spines are in a different data center. If both leafs lose connection to the spines and controllers in that data center, how does vPC operate? Because according to my understanding there is no direct peer connection between leafs in ACI mode, it uses the fabric for its keepalives. But what if that goes down?

[dlots]

It would act the same way it would if all it's uplinks were removed... because that's what has happened if it looses it's spine connection.  If it's setup right I would think it would give up all responsibilities since it would loose all it's routes and litterly everything.  Never tried it but that's what I think would happen.

[icecream-guy]

I hope you didn't design that network.

white paper might help
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-740861.html

[LynK]

I hope you didn't design that network.

white paper might help
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-740861.html

In the paper you sent me, which I was referencing it states:

"ACI does not require the use of a vPC peer-link between leaf nodes"

"When Remote Leaves are configured as a vPC pair, they will establish a vPC control plane session through over the upstream router"

We have dark fiber, and MPLS with which we are looking at designing remote leafs. I brought up to cisco what happened if both dark fiber and mpls went down in a DR scenario? Would the VPC pair go split-brain and then think all ports are orphaned, and then disable all ports? They are looking into it.

[Otanx]

What is the use case of a vPC pair at separate data centers? The only reason I have done vPC is to dual home LACP servers.

-Otanx

[wintermute000]

The white paper is quite clear...

This is assuming you have configured local L3out.

"When Remote leaf switches are configured with vPC domain and end host/L3out is connected to Remote leaf switches with vPC. Following behavior will be seen during 2nd (IP Network) or 3rd (all Spines) failure.
●   Traffic to known end points/prefixes connected to Remote leaf will continue to work.
●   Traffic to destinations outside of Remote leaf pair will drop since Spine is not available.
●   No configuration can be added to Remote Leaves since connectivity to APIC is lost."

Enjoy learning the dark arts of ACI. I would caution you against going all-in just for this feature. For example, multi-site EVPN pseudo-BGW is pretty much this, except in the VXLAN EVPN universe. NSX is even better, doesn't care where the hosts are as long as there is sufficient IP connectivity/MTU/BW. I'm assuming you're approaching ACI green as you've never previously mentioned this technology before. If I'm wrong I apologise.

Either way, extensions are pretty much designed for temporary migration scenarios, you're basically extending your failure domain, its entirely dependent upon the control plane at your 'native' DC.

[LynK]

@winter

100% green, first implementation. We were looking at NX-OS and ACI options for VXLAN (and NSX). We are using remote leaf because the cost savings of not having spines at each site's data centers. We believe that between our dark fiber, and MPLS we should be safe. But we were wondering in the event of both going offline, I fail to see how vPC will work when the peer links go offline because they are tied into the fabric that is now completely isolated. In NX-OS this would not be an issue because there are dedicated linked between VTEPS. But in ACI vPC are no dedicated links between leafs for vPC.

@Otanx,

I think you mean having a vPC pair stretched across DCs, this is not what I mean. I am talking about 2 leafs in a small datacenter B, which tie back to 2 spines in a larger DC A. The issue is, is that in ACI the heartbeats for vPC run over the fabric, and there are no dedicated links between the two vPC hosts. Therefore if they lose access to the fabric, they will continue to run (obviously), but how will vPC work if the keepalives goes through the fabric become offline. I'm guessing split-brain with orphaned ports that do not go anywhere. This is not an issue in NX-OS because in NX-OS your leafs have dedicated links between each other for vPC heartbeats.

[Otanx]

Ah, OK. I understand now. Don't have an answer for you, but I see how that could be a problem.

-Otanx

[wintermute000]

ACcording to the white paper, the vPC 'control plane' simply uses the uplink routed path. There must be limited local intelligence even if the APIC is cut off which keeps the vPC construct functional, but you can't get out of the remote leaf pair, it effectively becomes an isolated local vPC pair.


If standards based VXLAN EVPN and a BGP control plane is more your jam, suggest you look at this. I have successfully PoCed this to hell and back (for a tier-1 customer) and it 100% works. Its actually very similar to ACI remote leaf under the hood, and it was out on the market before that (had it going as far back as Jan).


https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html#_Toc498025694


Either way I would caution against viewing these remote leaf extension technologies as anything other than a stopgap or migration tool. If the site is small and won't warrant its own ACI cluster then I would question employing a technology that requires 3x control nodes at each site, there's this thing called a MP-BGP control plane where you don't need any controllers (well, unless you run iBGP + RR and you view the RRs as control nodes)

[LynK]

@winter

thanks for the response. I understand VXLAN EVPN quite well, and we are considering it an option. But we are also consider EVPN. Can you explain the "isolated local vPC pair" if you can, because that is generic.

If you have a vPC pair, with a peer link and keepalive failure, this is what happens in NX-OS:

"
vPC Peer Link Failure Followed by a Peer Keepalive Link Failure
If a peer link failure occurs, the vPC secondary switch checks if the primary switch is alive. The secondary switch suspends its vPC member ports after it confirms that the primary switch is up.

vPC Keepalive Link Failure Followed by a Peer Link Failure
If the vPC keepalive link fails first and then a peer link fails, the vPC secondary switch assumes the primary switch role and keeps its vPC member ports up.

If the peer link and keepalive link fails, there could be a chance that both vPC switches are healthy and the failure occurs because of a connectivity issue between the switches. In this situation, both vPC switches claim the primary switch role and keep the vPC member ports up. This situation is known as a split-brain scenario. Because the peer link is no longer available, the two vPC switches cannot synchronize the unicast MAC address and the IGMP group and therefore they cannot maintain the complete unicast and multicast forwarding table. This situation is rare.
"

AKA causing suspended ports, or loops, or something else. But there is no documentation proving this happens the same in ACI.

[wintermute000]

The white paper is quite clear, the vPC pair uses the normal underlay (upstream "router") for connectivity. I would guess that the L2 is encapsulated in VXLAN as normal and routed through the 'upstream router'. I am not an ACI guy.
Cisco are running around telling people that not needing a vPC peer-link is the dogs bollocks which I don't quite understand, its one freakin port saved.

You'll have to read that white paper again in depth and/or ask your friendly SE regarding what exactly happens when the remote vPC pair gets isolated, but the points I noted above are quite clear.

If I read you correctly you are confusing vPC behaviour upon loss of upstream router and vPC behaviour upon loss of connectivity to ACI fabric (which is what you keep talking about losing dark fibre+ MPLS). The two are not the same, your local routers may be fine, but all the WANs are down. The white paper again talks about both scenarios. Figure 32 illustrates clearly what happens when the remote leaf can't talk to each other, it brings everything vPC down.