Fun Project: Building Out a Global Network

[deanwebb]

Since most of us are in a holiday freeze and can't mess with anything at work, I thought it would be a fun project to design a new global network for a multinational that must also comply with PCI-DSS regulations.

The rules are simple: every one of us is an engineer or architect working for the company, we'll call it NFS Financial. This thread is essentially our design discussion where everything comes into consideration and we plan this out. If we have an argument, I would recommend that we could fork our discussion at that point and carry on with separate discussions, so that we can see why or why not to do certain things.

So, we start with the specifications:

NFS has 100,000 employees. Exactly. Lucky us for that. There are 25,000 in the main office in New York; 10,000 in San Francisco; 2000 each in London, Frankfurt, Hong Kong, Singapore, and Zurich; 5000 in the IT Centre in Hyderabad; another 5000 in the IT Centre in Sofia; and another 5000 in the IT Centre in Mexico City. Those are the big offices.

The other 40,000 users are in sales/brokerages around the world. There are 400 offices, each with 100 users.

Because the execs at NFS are sick and tired of hearing complaints about legacy crap on the network, they are insisting that the entire network be replaced. All the old gear was leased, so they'll let it expire. There will be brand new everything - datacenter facilities, gear, everything. Including MPLS and/or local Internet circuits.

Our job, as mentioned above, is to provision stuff. Because the executives are non-technical, we have to think of everything. Right now, the project outline looks like this:

I. Design the global network
II. Implement the global network
III. ? ? ?
IV. PROFIT!

If any of us have questions about user demand or needs, any admin or moderator is able to provide a definitive answer. I encourage admins and mods to make up stuff that reflects real-world problems, such as there are (X) users at a site, but the site will only pay for bandwidth sufficient for half the users... that leads to some interesting QoS discussions.

[icecream-guy]

That Hong Kong site and China's issues with VPN, that's gonna be a sticky one,  should probably start there, and figure out how to connect that office and the other offices, smaller ones via VPN most surely.

Service layout will be an issue, what services where? can we go into cloud formation for some of that?
most of the services, would probably be in one of the 3 IT centers. One IT centers probably should be designated as an IT DR center.
or do active/active all the way around.

[deanwebb]

With the above out of the way, I suppose we should start with some requirements that came in from our CIO:

1. New York, San Francisco, Mexico City, Sofia, and Hyderabad need to have significant datacenter capacity. New York is the main site and San Francisco should be set up as a failover site. The other three IT centers should be able to back up critical functions, with the idea that each of the three IT centers would do about a third of the critical functions in the event of a massive failure.

2. Trading floors in NYC, SFO, Lon, HK, Sng, Fkf, and Zur need to have dedicated connections between all their peers. CIO asks for layer 2 adjacency because that's what the traders and database guys said was good.

3. Each local site will be responsible for its own Internet connection, so it needs to be affordable. Same for the gear supporting each of these 100-person sites.

4. The larger sites will have corporate deep pocket funding, so they can be pretty sweet. Don't go totally nuts, but you can get some nice stuff.

[deanwebb]

That Hong Kong site and China's issues with VPN, that's gonna be a sticky one,  should probably start there, and figure out how to connect that office and the other offices, smaller ones via VPN most surely.

Service layout will be an issue, what services where? can we go into cloud formation for some of that?
most of the services, would probably be in one of the 3 IT centers. One IT centers probably should be designated as an IT DR center.
or do active/active all the way around.

Director of Connectivity agrees, China needs to be a special use case. He'd also like protections between China and the rest of the world - as well as protections *between* sites in China, as there have been some odd issues with branches competing with each other.

For DR, see CIO comments, above.

Services include lots of data, lots of financial data, lots of personally-identifiable information (PII) data, customer-facing websites, backup traffic, VoIP (CIO wants to go with soft phones, globally), video conferencing between all sites over 100 employees and with video feeds available within the intranet, and they'd also like some security, please.

[icecream-guy]

Hardware requirements, Quotes...PO's..and purchasing need to be completed with this years funding.. you've got 4 days left.
LoL


JK

[deanwebb]

Hardware requirements, Quotes...PO's..and purchasing need to be completed with this years funding.. you've got 4 days left.
LoL


JK

Indeed, that's a funny one.

But, seriously... we need those numbers by the end of Q1.



In other regards, what is the best way to link up the 10 main sites? I'm guessing the smaller sites would be DMVPN via their local Internet provider. What are the options for hooking up the 10 big sites? And how does the full mesh requirement impact the design?

I can see provisioning local Internet for sites in different nations can wind up as its own entire thread... but fun to research and comment on.

In thinking for the 400 small sites, I thought about IP address allocation... each site would need a voice VLAN, a wireless VLAN, a guest wireless capability, some wired VLAN for IT infrastructure if nothing else, and some IPs for the WAN connections. Should we make those sites wireless-only for end-user devices? If so, how do we do that?

[icecream-guy]

at least 10Ge links to each of the larger offices, and fractional 10GE  guessing 3-5Ge for the smaller,
smallest sites with a DS-3,  or fractional  DS-3 maybe 5-15Mbps.

if the services are not centralized, then  maybe less.

DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.

[deanwebb]

DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.

So we're looking at hub-and-spoke VPN setups?

[icecream-guy]

DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.

So we're looking at hub-and-spoke VPN setups?

for the 400 sites, yes, I think single links (VPN tunnels) with backup tunnels into one of the other data centers for redundancy.

[deanwebb]

for the 400 sites, yes, I think single links (VPN tunnels) with backup tunnels into one of the other data centers for redundancy.

So that would be 3 links per small site. One link to NYC, one to SFO, one to Hyd/Sof/MxC, by region (APAC/EMEA/Americas).

That also means some big iron VPN hardware in NYC and SFO and a device with 1/3rd the capacity in each IT Centre.

I'm thinking we'd want something that can scale well and doesn't require manual entry tasks for maintaining each VPN.

[wintermute000]

Sorry guys, that's 2012 thinking.

One word: Viptela.

I would also accept Velocloud.

Dual internet at all spokes. Zscaler for outbound internet security and local internet breakout w/out maintaining 400 firewalls. SD-WAN vendor of choice will basically flatten it all out into one giant DMVPN with app-aware traffic engineering and local internet breakout to suit.

Given the complexities of routing between a global flat DMVPN and the core network for trading, Viptela seems like an easy answer, Velo has much less nerd knobs and critically does not enforce FVRF which means you have to worry about recursive routing and filtering.

I'd basically construct the large core sites/DCs into one WAN, then hang all branches off it. There isn't really the concept of a 'hub' as you'd just go to the nearest 'core' site, except with Viptela magic all core sites can basically act as backups to each other. Again no traditional redistribution filtering to worry about, you'd do it all in Viptela policy.


If we really want to stick with last decade technology/topoogy then sorry, dual DMVPN using iBGP with iBGP next-hop-self feature on the hubs is the only sane answer for scaling 400+ sites. Then after that its (relatively) straightforwards routing with a global eBGP core each one hosting regional DMVPNs. 

[icecream-guy]

Never heard of Viptela,   you had me interested until I saw this....

[icecream-guy]

Never heard of Viptela,   you had me interested until I saw this....

but could reduce costs from about 86 million traditional to 24 million..a 72% cost reduction.

[deanwebb]

Sorry guys, that's 2012 thinking.

One word: Viptela.

I would also accept Velocloud.

Dual internet at all spokes. Zscaler for outbound internet security and local internet breakout w/out maintaining 400 firewalls. SD-WAN vendor of choice will basically flatten it all out into one giant DMVPN with app-aware traffic engineering and local internet breakout to suit.

Given the complexities of routing between a global flat DMVPN and the core network for trading, Viptela seems like an easy answer, Velo has much less nerd knobs and critically does not enforce FVRF which means you have to worry about recursive routing and filtering.

I'd basically construct the large core sites/DCs into one WAN, then hang all branches off it. There isn't really the concept of a 'hub' as you'd just go to the nearest 'core' site, except with Viptela magic all core sites can basically act as backups to each other. Again no traditional redistribution filtering to worry about, you'd do it all in Viptela policy.


If we really want to stick with last decade technology/topoogy then sorry, dual DMVPN using iBGP with iBGP next-hop-self feature on the hubs is the only sane answer for scaling 400+ sites. Then after that its (relatively) straightforwards routing with a global eBGP core each one hosting regional DMVPNs. 

We don't want to stick with last decade technology, that's for sure! A big part of the reason behind this exercise in my mind is to provide a sort of design lab, where we can learn about tech that is outside of our normal comfort areas.

So, back to this Viptela... part of architecture is to define the solution and then let the engineers select the technology. In that sense, what are the competitors to Viptela? You mentioned Velocloud, are there others?

[icecream-guy]

So, back to this Viptela... part of architecture is to define the solution and then let the engineers select the technology. In that sense, what are the competitors to Viptela? You mentioned Velocloud, are there others?

They are both SD-WAN concepts, here is a link to a list of SD-WAN competitors

http://packetpushers.net/virtual-toolbox/list-sd-wan-vendors/