Interesting DCI Problem

[LynK]

Gentlemen,


I have a problem, and I am wondering if this would work.

The problem I have is I have two data-centers with layer 3 links between them. Each DC has its own ISP, but they are using different public IP address spaces at each data center. We are on a waiting list to get our own public IP addresses... the pain... it hurts.

The question I have is on the interim. How can I make this work. What I want to do is advertise the same carrier owned IP block at both sites, but prevent asymmetrical data flows. The only way I can think of doing this is putting the public IP address range on a vlan, and stretch it over the datacenter so our virtual firewalls have the IP block at both sites for failover. The problem is we have layer 3 connections.

My question for you is this. Lets say my internet range is 66.66.66.0/24 vlan ID 666 at datacenter A. What would happen if I created a subinterface ethX/X.666 and did encapsulation dot1q 666 on the subinterface at both datacenters with no IP address (or the IP of the network). Would that VLAN then be stretched over the subinterface to the other datacenter? So then at datacenter B's vlan 999 I have an edge router in the 66.66.66.0/24 network. Would I be able to ping the 66.66.66.X host on the other side?

I am thinking this will work... but I have no way to verify.

[icecream-guy]

VXLAN??
L2TPv3??
EoMPLS??
Cisco OTV??

[Dieselboy]

How is the link terminated at each end? Does it go into a provider router or does the fibre or copper handoff connect directly into your equipment? How does it connect into your equipment?

What I am thinking is that the link might be layer 2 but your side is managing it through layer 3. For example, you could have two, layer 3 switches linked together with cat6 and you could do any of the following:
- layer 2 vlan trunks
- routed ports (layer 3 only)
- layer 3 via SVI (layer 2, but terminates to a layer 3 virtual interface)
- layer 3 svi with layer 2 VLANs in addition
- you could even have routed port at one side and an SVI at the other. But with this, you wont be able to send a layer 2 vlan down there until the routed port is moved to SVI. Plus inconsistent config..

[spanning tree]
I've seen some providers drop spanning tree packets and UDLD packets while the data path was working fine. I expect their equipment has a bug and was intercepting those packets sometimes. They never accepted any "issue" and so every few months UDLD would shut down the link.

If it's layer 3 between provider router and your equipment then I think you can do gre but might be iffy, I've never seen layer 2 sent like this but I looked into it recently in case I needed to do it over an ipsec VPN.

[deanwebb]

Don't use VLAN 666. That's the one NAC uses to dump unauthorized hosts into.

[Dieselboy]

Evil VLAN

[Otanx]

You want to advertise the IP Space you use with ISP A out ISP B and ISP B space out ISP A? If you talk BGP to both of them it isn't hard. There are a few items you need to verify/do first. The first is to find out if your ISP advertises your space or if they advertise an aggregate that includes your space. If they advertise an aggregate then you will need them to advertise your space separately as well. Check looking glasses like bgp.he.net or others to see what they see. Then if you have a halfway decent ISP you can't just advertise anything. You will need to get a letter from the ISP owning the space that you can advertise it. This could just be a letter from the ISP, or they can SWIP the space to you so when people look it up in whois it says you own it. Once that is all done you can advertise the space. However, you have the issue you don't know which way the traffic will come into the data centers. To solve this just prepend 3 or 4 AS to your advertisement you want as the backup. You will still get some asymetric routing depending on your ISPs, but the majority of the traffic will go the way you want.

-Otanx

[icecream-guy]

Don't use VLAN 666. That's the one NAC uses to dump unauthorized hosts into.

use VLAN 1313.....

[LynK]

the link between the two nexus's is a point to point dark fiber link that we have running as a layer 3 link with OSPF.

The reason why I cannot just advertise both without a link between them is because it would bring a discontinuous network. What if I made an SVI for the vlan interace and make a subinterface with encapuslation for that vlan, but no IP assigned on the subinterface? would it broadcast over subinterface to the other datacenter?


We are going to be going with a VXLAN/NSX design in the future. The problem is I am knee deep in a firewall design right now.

[wintermute000]

Before you even get into this, I've never heard of an ISP allowing a block that they own to be advertised out of any other ISP.


As for the design question, my solution would be to build a L3 segment purely for routing in front of the FWs. Think of it as a normalisation layer. You can then route however you want within this segment, but as long as its symmetrical to the FW layer its all gravy. You should be able to just carve out a VLAN/subintf off your existing L3 links, or if not, use a tunnel. I've done this before and it works really well. You can even get crazy and have different public IPs at Site A vs Site B but NAT to the same load balancer on the inside so it doesn't matter which side it comes in on.

At the same time I'd be pushing hard for GSLB etc. so you can divorce IP from the app and do it properly (via DNS....) so it doesn't matter, then you don't need IP failover, just GSLB/DNS level failover which is what ALL the big boys do and is the correct AWS design pattern yada yada

[icecream-guy]

Before you even get into this, I've never heard of an ISP allowing a block that they own to be advertised out of any other ISP.

I have, but the two ISP's need to have a peering agreement in place.  not something one can do by themselves.

[Otanx]

Before you even get into this, I've never heard of an ISP allowing a block that they own to be advertised out of any other ISP.

We do this. The trick is to have the ISP "SWIP" the space to you. Basically this is the ISP telling ARIN/RIPE/etc that the customer is responsible for the space, but the ISP still owns it. The other requirements are that you have your own ASN, that the space you are working with is a /24 or larger, and that the ISP is forwarding your advertisement. Some ISPs will just advertise the aggregate that they own, and not your specific advertisement.

We have one ISP we are doing this with, and had no problems. The other ISPs were good as long as the routing DBs, and ARIN listed us and our ASN.

-Otanx

[wintermute000]

"The other ISPs were good as long as the routing DBs, and ARIN listed us and our ASN."

Cool, thanks for that. I wasn't actually aware ISPs were happy to do this.

[LynK]

Okay guys,

So here is a visio.

Things I have tried (got a gns3 server and attempted on 9000v's)

1) sub interfaces with encapsulation dot1q 666 and no IP (just IP on int vlan 666). - DOES NOT WORK
2) sub interfaces configured with a VRF and running ospf on 66.66.66.0/24 network on both sides and DCI links with private IPs - DOES NOT WORK - because each side "knows" about the /24 network, causes routing to fail
3) removing the sub-interface private IPs I created in step 2 and try to put 66.66.66.0/24 IPs in there. The problem with this is that I will only be able to use 1 interface... which removes resiliency. - DOES NOT WORK.

Im out of ideas... I think a tunnel might work (mentioned above).. I might give that a try too.

[Otanx]

Can you just split the space to two /25s one on each side and route using item 2? What is your final goal? If you really need the L2 spanned between the two you can always take two of the fibers from your current link, and connect them to access ports on both sides.

-Otanx

[LynK]

Splitting will not work unforurnately because we want to be able to properly failover. Looks like our options are

1) move a link or two (or buy 2) for L2 extension

2) VXLAN