adding resiliency to the company wan

[scottsee]

it's been a few years since I've built one of these out.. here is the scoop.

Clarified post with Visio in next post

currently
5 sites
40mb MoE for backend traffic and backups to the datacenter (centurylink)
10mb basic business class internet for default quad zero routing(centurylink)
Meraki networking on layer 2 and 3
no resiliency

new
5 sites (scalable for new acquisitions)
40mb MoE for all traffic - including internet (centurylink)
20mb hot standby (different carrier for real failover)
implement QoS over MoE
Cisco edge routers with IPSLA to monitor MoE statistics
2nd business internet carrier for IPSLA failover
all office are local - same CO

normal - pass all traffic through site firewall, router, MoE, datacenter firewall, 1gb internet access for all sites through Datacenter
failover - pass all traffic through site firewall, router, 2nd carrier to datacenter..

tricky part... our lync phone system and SBC are at the datacenter, with a failover route to a hot/standby SBC at our corporate office. I don't even want to go there right now.. Data first. 

we have 5 x RFC 1918 contagious internal ip spaces- so I'd need to nat that traffic and pass it out the failover circuit directly to the required site, or to the datacenter depending on the conditions of the other moe links being up or down. Can I have IPsla monitors at all MoE sites and have it report statistics of the remaining WAN to let my on premise site router know what route would is more efficient.. ugg my routing is shit!!! I think that's a IPsec / OSPF tunnel right? hummm.

Right now 2 sites have 3570 switches passing up to a single Meraki MX100. Maybe GLBP on those sites instead of an edge device? I need to brush up on FHRP and whats possible.

last 3 sites have Meraki 220 cores passing up to Meraki Mx100's. I'm thinking the simplest way to enable failover routing at these sites is to add an edge router like a 2911 and place them in front of the Meraki MX and monitor the MoE circuits..

Right now everything is on static routs.. maybe time to deploy OSPF?.. thoughts?

[wintermute000]

CBF interpreting all that... it may make sense to you or anyone that's worked on your network but to an outsider I'm afraid I need a diagram

e.g.

WTH is MoE
Is that 40M MoE per site or aggregation
WTF is centurylink
What's 'same co'


I have a rough picture but its just easier to ask for a diag

[scottsee]

CBF interpreting all that... it may make sense to you or anyone that's worked on your network but to an outsider I'm afraid I need a diagram

e.g.

WTH is MoE
Is that 40M MoE per site or aggregation
WTF is centurylink
What's 'same co'

I have a rough picture but its just easier to ask for a diag

MoE = Metropolitan Ethernet - Private WAN
CenturyLink = Major US ISP
Same CO - Central Office / Return ISP switching location

Sorry - Tried to eat breakfast and make a quick post.. here is a quick Visio of what I'm thinking..

• Each site has a Cisco Meraki MX100 firewall
• I looked and they don't do IPSLA
• They do support OSPF as of the latest firmware push
• New edge device updtream from the MX to handle routing and IPSLA at each site - not reflected on Visio
• Each site has roughly 250 - 750 devices
• Datacenter - 100'ish servers - single MX100 with Coldspare 2nd MX100 dirrently into Meraki 220 core switches..
• all sites are collapsed core design, 10gb trunks between switches
• I would love to implement FHRP, but I don't think the Meraki 220's have the feature - I think the 300 series might though which I can buy

I am also moving our VPN site design to the data center (separate thread) so I might put in a new edge device like the 5512x or Cisco router that can handle the OSPF and IPSec tunnels..

Any more clear? I'll get more granular and detailed on site design if needed..

[wintermute000]

1.) Is your MoE a layer 2 or layer 3 cloud? If latter you will have to discuss with your provider what your routing options are. OSPF is very rare in MPLS-VPN networks, I could go on for hours but I won't (TL:DR - it flat out sucks for PE-CE scenarios).

2.) Even assuming say MoE is layer 2 so you have end to end control, have you given any thought as to how your OSPF will work across both WANs? (hint: how are you doing your areas)?
- how are you doing the x-connect between the 2 WANs?
- have you scaled your routing design for sites that are VPN only in future and/or future additional WAN hub or VPN hub connections?
- have you read the literature on how OSPF is basically the worst possible choice (over BGP or EIGRP) over VPNs even if supported?
- strongly recommend you do not just shove everything into area 0 and hope for the best.

3.) Does your VPN solution even support OSPF? (hint: it will have to be encapsulated in GRE or mGRE more or less)

4.) Are you doing 'split tunnelling' when sites are using the backup VPN i.e. are you tunnelling internet traffic back through your DC even in failover.

Frankly speaking I would also look heavily into what the merakis can or cannot do, if OSPF is the 'latest firmware' then I would probably think twice about making them do a traditional ISR's job, at least not without some proof of concept/lab work.


I would also caveat my above comments with I don't know squat about Meraki, so maybe they have special sauce that fixes everything, or something like that

I love these green-fields design scenarios.... by any chance is your company is cool with bringing in randoms from dah interwebz as consultants

[javentre]

if OSPF is the 'latest firmware' then I would probably think twice about making them do a traditional ISR's job, at least not without some proof of concept/lab work.

Agreed.

[javentre]

Quick question.   Meraki is cloud controlled.  So, you configure OSPF over the cloud, and the config is pushed down.  That's how it normally works with the wireless stuff, and it sounds that way from reading a blog about their (OSPFv2) implementation.

How do you bring up a new router that can't yet talk to the cloud, because it needs OSPF to get the routes to get to the cloud?

[icecream-guy]

MPLS and MP-BGP, and let century link worry about the connectivity,  would kill the requirement for NAT. no NAT is good NAT.

are the failover ISP all the same ISP?  go MPLS there too.

[scottsee]

Had a meeting with our carrier reps. I'm going to "keep it simple stupid". I'm also going to do this in several phases.. First one is get thing migrated.

• bump layer 2 MoE upto 40mb at each facility
• add 2 x 5525x's at the core
• CenturyLink promises upto 5 DSCP marking available over our WAN.. Start with 2 - Web and Non-Web
• Change all of my quad zero next hops

I'll tackle the DSCP, dual carrier and IPSLA enabled devices after phase one.

[routerdork]

Quick question.   Meraki is cloud controlled.  So, you configure OSPF over the cloud, and the config is pushed down.  That's how it normally works with the wireless stuff, and it sounds that way from reading a blog about their (OSPFv2) implementation.

How do you bring up a new router that can't yet talk to the cloud, because it needs OSPF to get the routes to get to the cloud?

My guess is that at least on interface is DHCP enable and you've got to set this up using something with an internet connection. I'm not a fan of Meraki and how things are done but I do like the concept.

[LynK]

What also is good, is some MPLS companies allow you to configure backup links to their concentrators. Which allows you to have the full meshing of BGP.

just a thought we are doing this