Extensible fibre ring design

wintermute000 · October 13, 2015, 05:56:52 AM

Running through potential options for a scenario that's quite interesting - 3 sites linked together by point to point dark fibre in a triangular loop.
Each site would hence have two routers or switches (1 for each p2p connection). The customer is not a service provider but they do have multi-tenancy requirements. They do not currently use L2 extension but I don't want to close this option off either. The customer is also not an entirely Cisco shop so EIGRP or R-PVST+ etc. is out.

Now there are a range of options I'm considering, just wondering if you guys had any thoughts on the following:

1.) The easy way is to route it all with conventional routers, job done. The problem with doing this is that it eliminates the possibility of running L2 extension without using pseudowires or the like.

2.) The 'obvious' way to do it is to use multilayer or metro ethernet switches. Trunk them all up, use SVIs to route. You can still drag a VLAN across if your really want, best of both worlds. Only issue is how to deal with the ethernet core. An obvious answer is to use REP or the like - but I don't have field XP with this, nor do I completely understand how it would interact with the MST/RSTP facing back into each customer LAN site. Don't want to stretch a single STP instance across - would MST adequately 'buffer' each site's L2 domain via creating a common core IST?
- need to make MST region root - disruptive
- what if I mandate L3 connectivity into the core? But then I'm doing away with stretched VLANs. So why bother and just route it?
- MST means common VLANS on all and blocking, as opposed to nicely segmenting it into a collection of point to point VLANs
- How does REP work if I have a REP ring core, but I have switches facing my fibre switches running traditional STP/RSTP? I'm trying to do some reading on this but not really finding a lot of info.

3.) Create a MPLS core with OSPF, run BGP on everything in a full mesh (with 1 VRF in the MP-BGP lol). If they want a L2 extension, I can run a MPLS pseudowire point to point, no looping possible. Heck could scale it up to a full VPLS

- can't think of any downsides other than complexity and the need to buy MPLS capable devices - for example, my 'goto' 3850 option is out. I will need 1Gb capable (hence 2Gb license) ISR4451s at a much higher cost. And convince the customer to run BGP.
- er.... overkill???

Inclined towards option 3.) but that's just me.... any thoughts at this stage?

Note cost may well come into it and dictate using 2960X switches (ugh) or the like.... in which case better bone up on MSTP LOL

LynK · October 13, 2015, 01:27:44 PM

why over complicate things?

Two L3 switches at each site with L2 links, and SVIs. That is the way I would do it. Why are you worried about using a single STP instance? You are running in a low latency DF environment.

Or do L3 links. Absolutely no need to do option 3 in my opinion

that1guy15 · October 13, 2015, 03:33:02 PM

I dont like the stretched vlans over SVI approach. Im currently cleaning up a similar setup and its going to be a long painful process.

If you are already looking at the ASR 1000 why not throw OTV in the mix? VLANs can still exist in multiple locations but the failure domain is limited to each location.

Else yeah I like the MPLS approach. Make it painful to do.

wintermute000 · October 13, 2015, 03:33:16 PM

Re lynk: Because stretching a layer 2 fault domain across three sites is bad?

Re: that1guy, otv could be a good way to enable layer 2 stretch whilst keeping it layer three. Asr could blow the price out though. And max performance license on 4451 is 2gb.....

I might have a chat with brocade and arista as well [emoji13]

routerdork · October 13, 2015, 03:51:30 PM

My first thought when I saw this setup was an ASR 903. Not sure of the cost though, I haven't worked with them in over a year.

NetworkGroover · October 13, 2015, 03:56:48 PM

Quote from: wintermute000 on October 13, 2015, 03:33:16 PM
Re lynk: Because stretching a layer 2 fault domain across three sites is bad?

Re: that1guy, otv could be a good way to enable layer 2 stretch whilst keeping it layer three. Asr could blow the price out though. And max performance license on 4451 is 2gb.....

I might have a chat with brocade and arista as well [emoji13]

Yeah, ask Arista about DCI with VXLAN

Here's a paper about it (Older now, but still good): https://www.arista.com/assets/data/pdf/Whitepapers/Arista_Design_Guide_DCI_with_VXLAN.pdf

packetherder · October 13, 2015, 04:44:29 PM

I like option 1. I think availability and scale are better addressed at higher levels in the OSI. Yeah, I know that's nearly impossible for a lot of enterprises.

If you have to do L2 over a DCI, EVPN might be another option. Supposedly better than pseudowires, but I'm way out of my domain here. Just starting the uptake on it.

wintermute000 · October 13, 2015, 06:32:10 PM

cheers for the reading and the suggestions.
Its a relatively simple setup so a pseudowire should suffice as the DCI requirement will be simply point to point. If I go cisco routers, OTV is another valid option.

NetworkGroover · October 13, 2015, 06:57:12 PM

Quote from: wintermute000 on October 13, 2015, 06:32:10 PM
cheers for the reading and the suggestions.
Its a relatively simple setup so a pseudowire should suffice as the DCI requirement will be simply point to point. If I go cisco routers, OTV is another valid option.

Whatever you choose, good luck!

LynK · October 14, 2015, 07:49:21 AM

Quote from: wintermute000 on October 13, 2015, 03:33:16 PM
Re lynk: Because stretching a layer 2 fault domain across three sites is bad?

If you are running MST/R-PVST I do not see how this is even remotely an issue. If you could explain why, I would appreciate it wintermute. I would personally go with L3 links. Let the L3 processing do 90% of the job for you. Ive never implemented OTV personally, but I would go that route.

It ultimately comes down to... what is the company's infrastructure? Apps? All virtual? etc.etc.

Reggle · October 14, 2015, 12:14:01 PM

Quote from: LynK on October 14, 2015, 07:49:21 AMIf you are running MST/R-PVST I do not see how this is even remotely an issue.

I've seen RPVST melt down due to a bug once, and I've seen MST melt down once due to an incorrect vendor implementation. No matter how good the design, a failure domain is a failure domain.

NetworkGroover · October 14, 2015, 03:53:12 PM

Quote from: Reggle on October 14, 2015, 12:14:01 PM
Quote from: LynK on October 14, 2015, 07:49:21 AMIf you are running MST/R-PVST I do not see how this is even remotely an issue.
I've seen RPVST melt down due to a bug once, and I've seen MST melt down once due to an incorrect vendor implementation. No matter how good the design, a failure domain is a failure domain.

I have to agree with Reggle. I'd contain failure domains as much as possible, wherever I could.