VXLAN confusions help

[Dieselboy]

I was trying to set up VXLAN between 2 ASAs. I have this guide: https://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/interface-vxlan.pdf

Has anyone used this? I just tried to set up a test and lost access to the remote ASA (transparent). I realised I needed to adjust the vni1 interface so I deleted it, and lost the ssh session. I Think the error was I added it to a bvi. 

I am having a bit of a difficulty understanding the concept for this configuration around the VTEP and VNI interfaces.

In the doc the VTEP is a regular interface. Am I correct that the VTEP is like a VTI tunnel interface with the tunnel source / tunnel destination on say an IOS router?

Then there's the VNI interfaces. If I understand correctly, this is the interface that drops into the broadcast domain you want to stretch with vxlan. Although it also needs an IP address.

===

So, for the VTEP I was thinking to use the ASA outside interface which is also providing internet access and IPSEC VPN. So I configured my ASA backup interface as a VTEP and it stopped routing internet via that interface. Hmm. So I can configure another physical interface but thats where I started getting confused. Maybe a loopback would be better? But ASAs dont do loopbacks.

So then I thought, may be it would make more sense from the transparent side. So began working from that side back to here, then lost access. I read the docs again, and it says I dont use a bvi for the vtep. The confusion came about because the vni defaulted to bvi1, and I wanted to separate this to avoid any issue, so I configured another bvi and then deleted the vni1 I created before to start over.

The idea with this, is:

LAN1 -> ASA -----ipsec vpn / internet--------ASA -> LAN1

[wintermute000]

Yes your understanding is correct.
With normal leaf-spine networks the VTEP is in global and VNI is in tenant VRF. Dunno how that works in ASA

What's your use-case? I'd trust a L2TPv3 tunnel on ANY IOS device over a firewall trying to do VXLAN...

[Dieselboy]

Well ultimately I would like VM instances to fail over to another site and keep their layer 3 addressing, so that they dont have to be re-IP in a failover scenario.

[wintermute000]

How are you going to handle routing? Where's their default GW if the old site is kaput? What about when one VM moves but the rest of the subnet is still there?  How are both ASA's going to both present the same default GW at the same time?
How are you handling L2 loop prevention?
Any asymmetry in either direction and there goes anything stateful (FW, NAT, WanOp, visibility tooling, etc)
Stretched L2 is always presents design problems (depending on your exact use-case and scenario)

[icecream-guy]

ASA clustering?  Should help with most of these issues. in theory one can cluster ASA across DCs

[Dieselboy]

How are you going to handle routing? Where's their default GW if the old site is kaput? What about when one VM moves but the rest of the subnet is still there?  How are both ASA's going to both present the same default GW at the same time?
How are you handling L2 loop prevention?
Any asymmetry in either direction and there goes anything stateful (FW, NAT, WanOp, visibility tooling, etc)
Stretched L2 is always presents design problems (depending on your exact use-case and scenario)

Wo there cowboy - it's a test

The scenarios I am researching for are:

Scenario 1 - main office up, nothing at remote site.
Scenario 2 - main office outage, everything at remote site.
Failback is, shut down remote site. Bring up main site.

I'm not sure if the ASA has to be a default gateway for the subnet.. I had understood it that the ASA needs a layer2 link into the subnet, so that when a system ARPs for the other, the ASA will respond in proxy ARP fashion. My test was going to clear this up.

But before I can get to that stage, I need to make sure that the Openstack -> storage API is going to work at the remote site. I heard that Nimble Storage dropped support for Openstack in nimble OS 5.x. Our remote site has 5.x version. So in this 'test' I want to make sure that openstack can do the storage-y things on the remote array. I am hoping that 'dropped support' means that the APIs are still there and functional. Once I know enough, I can plan a path. I was going to test that the volume could be created.

[deanwebb]

Me security.

Me want static route on ASA. Hmm!