Same subnets (public and privates) across two datacenters

[kurdam]

Hi,
I'm contacting you because i'm facing a pretty complex problem and i'm a bit lost.
I'm working in a company that rents a bay in a datacenter and we are planning for some time to go big data with a second site to host our machines.
Our problem is that we would like to have our public IPs available on the first site as well as the second site.
I am wondering what technologies we can use to set this up.
Our main point of worrying is that we would like as much as possible to be able to keep on site A and B the same VLANs numbers and IPs (so that if we decide to take an hypervisor with all its machines and set it up on site B it will work without any modification in the configurations.
We are already planning on installing a site-to-site IPSec tunnel between the two datacenters in order to do live migrations (we have about 5ms of latency so in theory we can even have our storages on site A and our VMs on site B)
Some of the public IPs are provided by the DC itself but we also have a rank that we purchased at the RIPE. (we are owning it but we delegated the management to the DC1 so they are routing our IPs to us).
DC2 will also rent us some public IPs.

I'm joining a schema of our infrastructure so i can illustrate what i'm saying.
Based on this picture, in the first time we would like to empty Cluster 1 and ISCSI storage 1, move them to DC2, bring them back up with the same IPs and then be able to do a VMotion from site A to B with the VM keeping the same public IP.
We have also some nodes in Proxmox, my concern with the privates IPs between the 2 sites is concerning the Clustering of all these nodes (i know i probably can use NAT in order to "masquerade" the IP but i rather keep this setup as simple as humanely possible).
I'm having a hard time fining the documentation on somethin as specific as this. I asked around me already to friends that knows a lot more than me and they proposed me several ideas to achieve this:
- the first one told me that via the IPSEC tunnel (and some pretty specific routes) we can achieve this.
- another one told me that EBGP was the way to do it.
- and i found this thread : https://arstechnica.com/civis/viewtopic.php?f=10&t=1324975 that recommends more VRRPe and OTV.

Honestly, i'm a bit overwhelmed with the quantity of informations and i would like some advice on what you this is the best and easiest way to achieve this.

For our routers we are rocking Fortigates if that can help you.

Thank you in advance for your help and don't hesitate to ask me more informations if you need.

[icecream-guy]

Why not move it to the cloud and not worry about moving between datacenters?

[deanwebb]

^That, or use a temporary resting place on a different IP. Once everything is on the temp. IP, decommission the old IP in DC1 and then provision it in DC2, then move everything from temp storage to the DC2 location.

[kurdam]

We have too many servers in order to do that and it will take too long and require too many configurations in order to replaces every linkages for the IPs between the servers.
We are talking about 150 VPS, at least 8000 web servers and i'm not even talking about the DNS servers, the mailboxes ...
I think i'm going to go with VXLAN over IPsec
First by getting all my VLANs into a VXLAN
https://docs.fortinet.com/document/fortigate/6.2.0/new-features/392860/vlan-inside-vxlan
Than by making that VXLAN transit over a site to site IPSEC tunnel
https://docs.fortinet.com/document/fortigate/6.2.0/cookbook/821119/vxlan-over-ipsec-tunnel
I must find a way to do that networking-wise with a solution that will be totally transparent for the machines as well as for the customers.

[deanwebb]

With about 8200 servers, yes, I'd agree my plan would be dead in the water.

That does sound like a solution that will introduce some additional latency. My first question is if you have latency-sensitive apps/processes like a trading floor or control system? If so, they may need to be placed in a high-priority queue that would displace bulk traffic.

[kurdam]

Yes, we do have latency sensitive applications and connections between servers.
But that problem doesn't scare me too much because we intend on migrating these servers at the same time (or ASAP).
And our latency between DC1 and DC2 is around 5ms.
I went and tried to create a lab with this configuration in mind but honestly i'm pretty lost. 
I'm not used at all with working with these technologies.
I found a video on youtube of someone creating a lab like this one : https://www.youtube.com/watch?v=69oa55LsoAc but he only works with one VLAN and i hope it will be the same with multiple VLANs.
I really think that this method if feasible and is adapted for our needs. The only thing i'm not sure about is my ability to set up something this critical.

[deanwebb]

So you're looking at a large amount of work, you may want to get hold of your Fortigate account team and see what they recommend. There could be a way to do what you want with what you have, or you may need a bigger box.