MST question

[jericho]

Hi,

I've just been dropped into a DC environment where odd stuff is happening on the network. It looks like most of the issues are down to a VPC misconfig, but there is also some STP weirdness. They are running MSTP, which I've not got a great deal of experience with and my google skills appear to be failing me today...

The physical setup is 2 DCs with a VPC pair at each site. Each VPC peer has a link to the interconnecting WAN, which is L2. Each site has a number of local VLANs, with a handful that are stretched between the 2 sites. They currently only have 1 link at the secondary site active, as everytime  they bring it online, they get what looks like a broadcast storm on the WAN.

I think I can see the issue(s), but if anyone has any other pointers I'd be grateful.

Firstly, they have named the MST regions differently (they are named after the DC's city). I was under the impression that all switches participating in the same MSTP needed to have the same region name.

Secondly, should the native VLAN be left in instance 0?  I'm struggling to find a definitive answer on this other than a couple of old support forum posts which aren't overly clear (to me at least).

In this case it (VLAN 115) has been moved to instance 3, the instance for the stretched VLANs.

Site A config

Code Select Expand


spanning-tree mst 0-1, 3 priority 4096
spanning-tree mst configuration
  name DC1
  revision 1
  instance 1 vlan 101, 103, 105, 113, 117, 121, 131, 141, 145, 161, 191, 195
  instance 3 vlan 115-116, 191, 192, 193, 196, 197, 198, 199


Site B config

Code Select Expand


spanning-tree mst 0, 3 priority 12288
spanning-tree mst 2 priority 4096
spanning-tree mst configuration
  name DC-2
  revision 1
  instance 2 vlan 102, 104, 106, 114, 122, 132, 142, 146, 162, 192
  instance 3 vlan 115-116, 191, 192, 193, 196, 197, 198, 199


Thanks in advance

J.

[wintermute000]

For switches to be part of the same MST region, name must match... that config is clearly two different MST domains.

Rest of it is a bit hard to follow in text, its not clear whether your WAN links are the actual vPC (back to back vPC over 2x L2 WAN circuits?). But would need the rest of the configs.

Read this
http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000-series-switches/C07-572834-00_STDG_NX-OS_vPC_DG.pdf

[jericho]

Thanks,

That's probably why STP isn't working between the 2 sites, it's all supposed to be one region.

Each site has it's own VPC pair, with a (now) different domain ID. That was step 1 this morning and sorted out a lot of weirdness. The WAN links are standard trunk links with the stretched VLAN's permitted across them. There is a transit VLAN setup for inter site routing (VLAN 198) between the DC's.

The native VLAN is 115. That's where the routed access link for the remote customer sites is. I've suggested this isn't the best idea ever, but I'm stuck with it at the moment as the remote sites aren't under my, or my customers, configuration control. Whenever I've used MST in the past (rarely), I've left the native VLAN in instance 0. Here they have moved it into the same instance as the other stretched VLAN's.

Everything else was working when I left for the day. I've put it a request to change the region names so they match, I'm just unsure about which instance the native should be in.

Scrappy diagram attached if that helps.

Cheers

J

[LynK]

Jericho,

I would consider making your two L2 WAN links into a port-channel, or make them L3. STP will end up blocking one of the links if you configure them independently. If you are going to do stretched vlans across a layer 3 boundary, you are going to need some sort of DCI solution (vxlan most likely)

[jericho]

Jericho,

I would consider making your two L2 WAN links into a port-channel, or make them L3. STP will end up blocking one of the links if you configure them independently. If you are going to do stretched vlans across a layer 3 boundary, you are going to need some sort of DCI solution (vxlan most likely)

I'd love to. Unfortunately the WAN providers don't support that.

It's not stretching across a L3 boundary, the WAN is L2. The L3 interface is a VLAN carried to a firewall.


It's not how I would have designed it if I'd had the choice, but today is my first involvement in this.

Cheers

J

[LynK]

Jericho,

I would consider making your two L2 WAN links into a port-channel, or make them L3. STP will end up blocking one of the links if you configure them independently. If you are going to do stretched vlans across a layer 3 boundary, you are going to need some sort of DCI solution (vxlan most likely)

I'd love to. Unfortunately the WAN providers don't support that.

It's not stretching across a L3 boundary, the WAN is L2. The L3 interface is a VLAN carried to a firewall.


It's not how I would have designed it if I'd had the choice, but today is my first involvement in this.

Cheers

J

You misunderstand what I was trying to say.

1) If you have 2 layer 2 links you are going to have 1 active-forwarding, and the other blocking. You currently have 2 layer 2 WAN links, and if you decide to keep them layer 2 you should etherchannel them so that you can use both in an active-active solution, and STP will not have to block one of the links.

2) You can also run layer 3 over your layer 2 links by removing switchport on the WAN interfaces and IP them and use ECMP with an IGP. I would do this, but make sure you also have a DCI solution if you are going to stretch L2 over L3.

[jericho]

You misunderstand what I was trying to say.

Probably, misunderstanding people is my superpower 

1) If you have 2 layer 2 links you are going to have 1 active-forwarding, and the other blocking. You currently have 2 layer 2 WAN links, and if you decide to keep them layer 2 you should etherchannel them so that you can use both in an active-active solution, and STP will not have to block one of the links.

Yep, unfortunately, the WAN provider(s) don't support MC-LAG, or anything useful like that, so I'm stuck with what they have.

2) You can also run layer 3 over your layer 2 links by removing switchport on the WAN interfaces and IP them and use ECMP with an IGP. I would do this, but make sure you also have a DCI solution if you are going to stretch L2 over L3.

That would have been what I would have suggested, but they don't have the licences (or expertise to be honest) to run OTV or anything like that, hence the L2 stretch.

There are a whole host of things in this environment that don't sit quite right (native VLAN being used as a transit, reliance on remote customer sites not "misconfiguring" their equipment and breaching the security boundary etc) but it's what I have to work with. A suggestion of a complete redesign now isn't going to even reach the end customer.

Current plan is to get the region names to match and, depending on the outcome of that, possibly move the native VLAN to instance 0.

Thanks for the help & suggestions.

Cheers

J

[LynK]

@jericho,

Watch out for the native vlan issues, they can cause a whole assortment of issues. I would recommend proceeding with your plans.

But I am failing to understand why you cannot do an etherchannel over L2 WAN links. It should not be an requirement for any WAN providers. You should be able to etherchannel over any L2 link, from SW1 to SW2 without any issue (that I am aware of).

What is stopping you from doing the attached?

[wintermute000]

If its VPLS, provider is likely dropping LACP. Come to think of it you'd need to explicitly allow STP as well if I recall my SP stuff correctly.
If its straight DWDM then its fully transparent.
If he runs a vPC there are routing issues (no routing adjacency over vPC - the infamous Brad Hedlund article) unless you are game to go to bleeding edge 7.x release.

[jericho]

Yep, the previous guy had to get the provider to permit STP as they've had STP issues since day 1.

They are on 7.3.1. Routing between the local site subjects seems OK, but there is some odd HSRP behaviour.

At the moment the only link between the vPC peers are the peer-link & keepalive. I'm sure I've seen a suggestion that there should be another for L3 & HSRP type traffic but I can't find it now.

I can see how EC would be beneficial if it was just the two DCs, but how would that affect the client sites also connected to the WAN? In my head EC is for a direct link between 2 nodes. Admittedly it's been a while since I was hands on with kit & the NX stuff is all pretty new to me as well.

Cheers

J








Sent from my iPhone using Tapatalk Pro

[wintermute000]

Its been awhile, but classic Nexus design MOST DEFINITELY recommends separating out Layer-3 routed links. Just go through all the best practices / design guides.
However, SVI on dedicated vlan via peer-link is also supported, I'd always go with a separate L3 etherchannel if at all possible.
If you're hell bent on running raw STP over a VPLS (frankly speaking I'm surprised they let you - there's usually quite onerous MAC address limits per VPLS instance, e.g. 1k or something like that) then I would suggest running a debug first to confirm you are seeing BPDUs etc. because if the STP is not passing properly then that explains why opening up the second link foobars the lot LOL

[LynK]

@winter,

I would only ever go the direct L3 link. I would try to stay away from SVI.

Good Article:https://supportforums.cisco.com/document/84446/nexus-vpc-recommendations#vPC_peer-link_recommendations



@jericho,

It depends on your setup. Your documentation is kind of confusing because you have two L2 links, then you have your WAN connections coming in across those L2 links in L3 (which makes no sense).

Lets say for example, than you have you WAN HUB in one of your data-centers, and you have a L2 etherchannel between DCs. What is going to happen is DC2 traffic goes L2 -> DC 1, then L3 from DC1 to your WAN HUB. Then it is all traditional L3.

If you are doing VPLS like wintermute said, then that is going to get even more interesting. What I would do is get two additional links, and have them go to two routers, and have that be your WAN edge. This would be good because you could have 1 hub router in each site. Let me draw a design for you.

See attached for the Design.

[NetworkGroover]

This was too long for me to read, but a common mistake folks make with MST is understanding the concept that EVERYTHING in MST configuration must match or else two switches will consider themselves to be in different MST regions and will cause all redundant links but one (no matter how you carve up your VLANs) to be blocked - the key indicator something is wrong (If you're expecting all switches to be in the same MST region), is the presence of Boundary ports.

Region name has to match, revision number, VLAN-to-MSTI mapping - all of it.  A hash is created from this and if the hashes don't match - boom - different MST regions.

Shame.. thought I did a blog write-up on this but I don't see it now... I definitely did an in-depth research and testing exercise and documented the results, but that was for a former job and it wouldn't have been proper for me to keep that documentation after I left.

[wintermute000]

I'm pretty sure the poor sod has L2 directly running from his VPLS through his Nexus and he can't partition it off @ L3 without a major WAN and/or mass CE changes. WHich BTW should be what you should aim for long term. VPLS is not designed to let you used it as a giant user access switch - its designed to let you run your own routers directly adjacent i.e. control your own routing point to point, and easy multi-tenancy via VLANs and VRFs without need for carrier intervention + overlay that shiz with q-in-q if you really want. You could even run MPLS over it, if the carrier supported large enough MTU for your label stack.


If you're terminating customer connections over it through your nexus, you need to move your PE to plumb directly onto the VPLS. Then you terminate it at layer-3 boundary and all your problems go away. You can have failover routers .1 .2 .3 .4 at each VPLS connection point no worries and then route between them separately (even via inception style layering more subinterfaces and VLANs, possibly VRFed off).

He wouldn't be the first person to fall into the L2VPN trap of not doing L3 from day one then having to re-learn the limits of layer-2  

I used to run 3x VPLS networks concurrently for a SP. Any layer-2 we let creep into the network would turn into a total PITA and we had to use L3 for any kind of routing control or traffic engineering policy (let alone WAN-wide shared blast domains). We ended up with an ironclad router-only policy. Never had an issue.