Since most of us are in a holiday freeze and can't mess with anything at work, I thought it would be a fun project to design a new global network for a multinational that must also comply with PCI-DSS regulations.
The rules are simple: every one of us is an engineer or architect working for the company, we'll call it NFS Financial. This thread is essentially our design discussion where everything comes into consideration and we plan this out. If we have an argument, I would recommend that we could fork our discussion at that point and carry on with separate discussions, so that we can see why or why not to do certain things.
So, we start with the specifications:
NFS has 100,000 employees. Exactly. Lucky us for that. There are 25,000 in the main office in New York; 10,000 in San Francisco; 2000 each in London, Frankfurt, Hong Kong, Singapore, and Zurich; 5000 in the IT Centre in Hyderabad; another 5000 in the IT Centre in Sofia; and another 5000 in the IT Centre in Mexico City. Those are the big offices.
The other 40,000 users are in sales/brokerages around the world. There are 400 offices, each with 100 users.
Because the execs at NFS are sick and tired of hearing complaints about legacy crap on the network, they are insisting that the entire network be replaced. All the old gear was leased, so they'll let it expire. There will be brand new everything - datacenter facilities, gear, everything. Including MPLS and/or local Internet circuits.
Our job, as mentioned above, is to provision stuff. Because the executives are non-technical, we have to think of everything. Right now, the project outline looks like this:
I. Design the global network
II. Implement the global network
III. ? ? ?
IV. PROFIT!
If any of us have questions about user demand or needs, any admin or moderator is able to provide a definitive answer. I encourage admins and mods to make up stuff that reflects real-world problems, such as there are (X) users at a site, but the site will only pay for bandwidth sufficient for half the users... that leads to some interesting QoS discussions.
That Hong Kong site and China's issues with VPN, that's gonna be a sticky one, should probably start there, and figure out how to connect that office and the other offices, smaller ones via VPN most surely.
Service layout will be an issue, what services where? can we go into cloud formation for some of that?
most of the services, would probably be in one of the 3 IT centers. One IT centers probably should be designated as an IT DR center.
or do active/active all the way around.
With the above out of the way, I suppose we should start with some requirements that came in from our CIO:
1. New York, San Francisco, Mexico City, Sofia, and Hyderabad need to have significant datacenter capacity. New York is the main site and San Francisco should be set up as a failover site. The other three IT centers should be able to back up critical functions, with the idea that each of the three IT centers would do about a third of the critical functions in the event of a massive failure.
2. Trading floors in NYC, SFO, Lon, HK, Sng, Fkf, and Zur need to have dedicated connections between all their peers. CIO asks for layer 2 adjacency because that's what the traders and database guys said was good.
3. Each local site will be responsible for its own Internet connection, so it needs to be affordable. Same for the gear supporting each of these 100-person sites.
4. The larger sites will have corporate deep pocket funding, so they can be pretty sweet. Don't go totally nuts, but you can get some nice stuff.
Quote from: ristau5741 on December 27, 2017, 09:10:28 AM
That Hong Kong site and China's issues with VPN, that's gonna be a sticky one, should probably start there, and figure out how to connect that office and the other offices, smaller ones via VPN most surely.
Service layout will be an issue, what services where? can we go into cloud formation for some of that?
most of the services, would probably be in one of the 3 IT centers. One IT centers probably should be designated as an IT DR center.
or do active/active all the way around.
Director of Connectivity agrees, China needs to be a special use case. He'd also like protections between China and the rest of the world - as well as protections *between* sites in China, as there have been some odd issues with branches competing with each other.
For DR, see CIO comments, above.
Services include lots of data, lots of financial data, lots of personally-identifiable information (PII) data, customer-facing websites, backup traffic, VoIP (CIO wants to go with soft phones, globally), video conferencing between all sites over 100 employees and with video feeds available within the intranet, and they'd also like some security, please.
Hardware requirements, Quotes...PO's..and purchasing need to be completed with this years funding.. you've got 4 days left.
LoL
:lol:
JK
Quote from: ristau5741 on December 27, 2017, 09:37:11 AM
Hardware requirements, Quotes...PO's..and purchasing need to be completed with this years funding.. you've got 4 days left.
LoL
:lol:
JK
Indeed, that's a funny one.
But, seriously... we need those numbers by the end of Q1.
:facepalm1:
In other regards, what is the best way to link up the 10 main sites? I'm guessing the smaller sites would be DMVPN via their local Internet provider. What are the options for hooking up the 10 big sites? And how does the full mesh requirement impact the design?
I can see provisioning local Internet for sites in different nations can wind up as its own entire thread... but fun to research and comment on.
In thinking for the 400 small sites, I thought about IP address allocation... each site would need a voice VLAN, a wireless VLAN, a guest wireless capability, some wired VLAN for IT infrastructure if nothing else, and some IPs for the WAN connections. Should we make those sites wireless-only for end-user devices? If so, how do we do that?
at least 10Ge links to each of the larger offices, and fractional 10GE guessing 3-5Ge for the smaller,
smallest sites with a DS-3, or fractional DS-3 maybe 5-15Mbps.
if the services are not centralized, then maybe less.
DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.
Quote from: ristau5741 on December 27, 2017, 10:53:15 AM
DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.
So we're looking at hub-and-spoke VPN setups?
Quote from: deanwebb on December 27, 2017, 10:59:14 AM
Quote from: ristau5741 on December 27, 2017, 10:53:15 AM
DMVPN is only useful when sites need to communicate directly with each other. which I would guess not a requirement for the 400 smaller sites. so they wouldn't need DMVPN.
So we're looking at hub-and-spoke VPN setups?
for the 400 sites, yes, I think single links (VPN tunnels) with backup tunnels into one of the other data centers for redundancy.
Quote from: ristau5741 on December 27, 2017, 11:30:35 AM
for the 400 sites, yes, I think single links (VPN tunnels) with backup tunnels into one of the other data centers for redundancy.
So that would be 3 links per small site. One link to NYC, one to SFO, one to Hyd/Sof/MxC, by region (APAC/EMEA/Americas).
That also means some big iron VPN hardware in NYC and SFO and a device with 1/3rd the capacity in each IT Centre.
I'm thinking we'd want something that can scale well and doesn't require manual entry tasks for maintaining each VPN.
Sorry guys, that's 2012 thinking.
One word: Viptela.
I would also accept Velocloud.
Dual internet at all spokes. Zscaler for outbound internet security and local internet breakout w/out maintaining 400 firewalls. SD-WAN vendor of choice will basically flatten it all out into one giant DMVPN with app-aware traffic engineering and local internet breakout to suit.
Given the complexities of routing between a global flat DMVPN and the core network for trading, Viptela seems like an easy answer, Velo has much less nerd knobs and critically does not enforce FVRF which means you have to worry about recursive routing and filtering.
I'd basically construct the large core sites/DCs into one WAN, then hang all branches off it. There isn't really the concept of a 'hub' as you'd just go to the nearest 'core' site, except with Viptela magic all core sites can basically act as backups to each other. Again no traditional redistribution filtering to worry about, you'd do it all in Viptela policy.
If we really want to stick with last decade technology/topoogy then sorry, dual DMVPN using iBGP with iBGP next-hop-self feature on the hubs is the only sane answer for scaling 400+ sites. Then after that its (relatively) straightforwards routing with a global eBGP core each one hosting regional DMVPNs.
Never heard of Viptela, you had me interested until I saw this....
Quote from: ristau5741 on December 28, 2017, 06:10:14 AM
Never heard of Viptela, you had me interested until I saw this....
but could reduce costs from about 86 million traditional to 24 million..a 72% cost reduction.
Quote from: wintermute000 on December 27, 2017, 05:49:17 PM
Sorry guys, that's 2012 thinking.
One word: Viptela.
I would also accept Velocloud.
Dual internet at all spokes. Zscaler for outbound internet security and local internet breakout w/out maintaining 400 firewalls. SD-WAN vendor of choice will basically flatten it all out into one giant DMVPN with app-aware traffic engineering and local internet breakout to suit.
Given the complexities of routing between a global flat DMVPN and the core network for trading, Viptela seems like an easy answer, Velo has much less nerd knobs and critically does not enforce FVRF which means you have to worry about recursive routing and filtering.
I'd basically construct the large core sites/DCs into one WAN, then hang all branches off it. There isn't really the concept of a 'hub' as you'd just go to the nearest 'core' site, except with Viptela magic all core sites can basically act as backups to each other. Again no traditional redistribution filtering to worry about, you'd do it all in Viptela policy.
If we really want to stick with last decade technology/topoogy then sorry, dual DMVPN using iBGP with iBGP next-hop-self feature on the hubs is the only sane answer for scaling 400+ sites. Then after that its (relatively) straightforwards routing with a global eBGP core each one hosting regional DMVPNs.
We don't want to stick with last decade technology, that's for sure! A big part of the reason behind this exercise in my mind is to provide a sort of design lab, where we can learn about tech that is outside of our normal comfort areas.
So, back to this Viptela... part of architecture is to define the solution and then let the engineers select the technology. In that sense, what are the competitors to Viptela? You mentioned Velocloud, are there others?
Quote from: deanwebb on December 28, 2017, 08:00:36 AM
So, back to this Viptela... part of architecture is to define the solution and then let the engineers select the technology. In that sense, what are the competitors to Viptela? You mentioned Velocloud, are there others?
They are both SD-WAN concepts, here is a link to a list of SD-WAN competitors
http://packetpushers.net/virtual-toolbox/list-sd-wan-vendors/
Hey, I saw Elfiq in the list! I love their stuff! Great tech support, as well.
OK, so SD-WAN all around the world.
Except in China. We can't fling data to the Internet in China, since the gov't there doesn't like that. We can send it across an MPLS network, out Hong Kong, and then to Hyderabad IT Center / Singapore major brokerage center. How do we get that to play well with SD-WAN?
Please tell us more about the applications they are using?
Is e-mail hosted in the cloud or in the central datacenters?
Are we using Skype to communicate, with all the fancy video and screensharing features?
Still using fileshares on local file servers or is everything hosted on local/cloud Sharepoint?
Other stuff to consider:
By the way, I am volunteering for the vendor dinners once they get wind of this project!
Quote from: SimonV on December 28, 2017, 12:28:21 PM
Please tell us more about the applications they are using?
Is e-mail hosted in the cloud or in the central datacenters? O365 in Azure
Are we using Skype to communicate, with all the fancy video and screensharing features? Yes , and integrated with the VOIP soft phones
Still using fileshares on local file servers or is everything hosted on local/cloud Sharepoint? A mix, local file servers, with partially completed migration to cloud in AWS
Quote from: ristau5741 on December 28, 2017, 01:32:21 PM
Quote from: SimonV on December 28, 2017, 12:28:21 PM
Please tell us more about the applications they are using?
Is e-mail hosted in the cloud or in the central datacenters? O365 in Azure
Are we using Skype to communicate, with all the fancy video and screensharing features? Yes , and integrated with the VOIP soft phones
Still using fileshares on local file servers or is everything hosted on local/cloud Sharepoint? A mix, local file servers, with partially completed migration to cloud in AWS
I'll add that there are big concerns from the security team about PII in the cloud.
To be fair, the security team always has big concerns about *everything*, but we think they're actually justified in this case. So much so as to raise a formal risk assessment line-item about it. Attached is an article about how an AWS guy allowed "All Authenticated Users" to access data on every US household from the US Census, hosted by Experian. Yes, that meant *any* AWS user got to download all 37GB of the database before it got secured. Over 500 datapoints on every household, as well.
Therefore, Security is keeping a close watch on how we handle encrypting data in motion.
Quote from: ristau5741 on December 28, 2017, 06:10:14 AM
Never heard of Viptela, you had me interested until I saw this....
You should do some more homework. They were the hottest thing in SD-WAN, so hot that 6 months ago CSCO forked out ~650 million and acquired them. So yeah they're Cisco now, but they were acquired precisely because they were demolishing IWAN. Go check out some older packet pushers podcasts if you don't believe me.
12 months ago I was the biggest Arista/Viptela/NSX fanboy out there, but the 900 pound gorilla is turning around slowly....
Look into the technology, its basically a cross between LISP (routing based on locations not destination IP, and hey there's cloud management so the traditional LISP server chicken-and-egg issues go away) and MPLS (label separation) with mandatory FVRF, built in multi-tenancy, BFD monitoring of every point to point leg and using the cloud to abstract away the traditional computational scalability problem of IPSEC phase 2. The MPLS style labelling also gives you built in multi-tenancy (VRFs without having to worry about per-VRF routing config) as well as the ability to vary your topology per 'VRF' (e.g. have a multi-point corporate VRF and a hub-and-spoke PoS network both over the same single overlay).
Its incredibly elegant and foundationally streets ahead of everyone else (who are all basically doing PBR using a DMVPN type overlay to flatten the next hop topology). There's no other SD-WAN vendor with their level of maturity and footprint - 6000 site reference customers (!!!), multiple (>2) US banks etc. Their devs are all former CSCO/JNPR/ALU routing devs who got sick of being handcuffed (I heard that one of the key guys was the key guy behind the original DMVPN code stack). They're not gussied up SMB class crap or WANop / FW guys putting lipstick on a pig (*cough Ri***bed cough *Fort**et**)
Protip: in 6 months you'll be able to buy an ISR with a Viptela component on it, and in 12 months there will be complete migration of the full Viptela stack onto ISR hardware. IWAN is still being supported, but the writing is clearly on the wall. IWANv3 (using, ironically, LISP) is scrapped, no more feature development, its dead, Viptela killed it. They're also going to roll the management plane (currently cloud instances out of AWS) into the SD-Access controller (DNA center i.e. the replacement for APIC-EM).
re: China, no problem, we'll just have to run a traditional IP-VPN to the china site, Viptela can happily overlay over any transport whether private or public. You just need to make sure there is an underlay default route (internet) accessibility so the China CPE can hit the cloud controller. No you don't get the SD-WAN fancy load sharing/failover features (unless you want multiple WAN transports), but it still abstracts away your routing topology issues.
Here are some of the corporate locations
City Nation
Chongqing China
Shanghai China
Delhi India
Beijing China
Mumbai India
Lagos Nigeria
Karachi Pakistan
Dhaka Bangladesh
Guangzhou China
Istanbul Turkey
Tokyo Japan
Bengaluru India
Moscow Russia
São Paulo Brazil
Lahore Pakistan
Cairo Egypt
Kinshasa DR Congo
Jakarta Indonesia
Seoul Korea, South
Wenzhou China
Mexico City Mexico
Lima Peru
London United Kingdom
Xi'an China
Hyderabad India
Chennai India
New York City United States
Shenzhen China
Bangkok Thailand
Suzhou China
Nanjing China
Dongguan China
Tehran Iran
Quanzhou China
Shenyang China
Bogotá Colombia
Ho Chi Minh CityVietnam
Hong Kong China
Baghdad Iraq
Fuzhou China
Changsha China
Wuhan China
Tianjin China
Hanoi Vietnam
Rio de Janeiro Brazil
Qingdao China
Foshan China
Zunyi China
Santiago Chile
Riyadh Saudi Arabia
Ahmedabad India
Singapore Singapore
Shantou China
Ankara Turkey
Yangon Myanmar
Saint PetersburgRussia
Casablanca Morocco
Abidjan Ivory Coast
Chengdu China
Alexandria Egypt
Kolkata India
Surat India
Johannesburg South Africa
Dar es Salaam Tanzania
Shijiazhuang China
Harbin China
Giza Egypt
İzmir Turkey
Zhengzhou China
New Taipei City Taiwan
Los Angeles United States
Changchun China
Cape Town South Africa
Yokohama Japan
Khartoum Sudan
Guayaquil Ecuador
Hangzhou China
Xiamen China
Berlin Germany
Busan Korea, South
Ningbo China
Jeddah Saudi Arabia
Durban South Africa
Algiers Algeria
Kabul Afghanistan
Hefei China
Mashhad Iran
Pyongyang Korea, North
Madrid Spain
Faisalabad Pakistan
Baku Azerbaijan
Tangshan China
Ekurhuleni South Africa
Nairobi Kenya
Zhongshan China
Pune India
Addis Ababa Ethiopia
Jaipur India
Buenos Aires Argentina
Incheon Korea, South
Quezon City Philippines
Kiev Ukraine
Salvador Brazil
Rome Italy
Dubai United Arab Emirates
Luanda Angola
Lucknow India
Kaohsiung Taiwan
Kanpur India
Surabaya Indonesia
Taichung Taiwan
Basra Iraq
Toronto Canada
Taipei Taiwan
Chicago United States
Osaka Japan
Quito Ecuador
Chaozhou China
Fortaleza Brazil
Chittagong Bangladesh
Bandung Indonesia
Managua Nicaragua
Brasília Brazil
Belo Horizonte Brazil
Daegu Korea, South
Houston United States
Douala Cameroon
Medellin Colombia
Yaoundé Cameroon
Nagpur India
Cali Colombia
Tashkent Uzbekistan
Nagoya Japan
Isfahan Iran
Phnom Penh Cambodia
Kochi India
Paris France
Ouagadougou Burkina Faso
Lanzhou China
Kano Nigeria
Dalian China
Guatemala City Guatemala
Havana Cuba
Rawalpindi Pakistan
Medan Indonesia
Accra Ghana
Visakhapatnam India
Gujranwala Pakistan
Jinan China
Karaj Iran
Peshawar Pakistan
Minsk Belarus
Caracas Venezuela
Sana'a Yemen
Sapporo Japan
Tainan Taiwan
Bucharest Romania
Curitiba Brazil
Shiraz Iran
Vienna Austria
Brazzaville Congo Republic
Bhopal India
Hamburg Germany
Manila Philippines
Kuala Lumpur Malaysia
Maputo Mozambique
Budapest Hungary
Warsaw Poland
Lusaka Zambia
Kathmandu Nepal
Tabriz Iran
Hyderabad Pakistan
Palembang Indonesia
Almaty Kazakhstan
Tijuana Mexico
Patna India
Montreal Canada
Davao City Philippines
Harare Zimbabwe
Barcelona Spain
Maracaibo Venezuela
Caloocan Philippines
Philadelphia United States
Novosibirsk Russia
Phoenix United States
Bulawayo Zimbabwe
Oran Algeria
Semarang Indonesia
Recife Brazil
Kobe Japan
Daejeon Korea, South
Kampala Uganda
Kawasaki Japan
Guadalajara Mexico
Auckland New Zealand
Vijayawada India
Fukuoka Japan
Kwangju Korea, South
Porto Alegre Brazil
Kyoto Japan
San Antonio United States
Santa Cruz de la Sierra Bolivia
Munich Germany
Kharkiv Ukraine
Yekaterinburg Russia
San Diego United States
Barranquilla Colombia
Milan Italy
Ibadan Nigeria
Makassar Indonesia
Córdoba Argentina
Prague Czech Republic
Mandalay Myanmar
Dallas United States
Montevideo Uruguay
Nizhny Novgorod Russia
Abuja Nigeria
Calgary Canada
Saitama Japan
Hiroshima Japan
Rosario Argentina
Brisbane Australia
Belgrade Serbia
Campinas Brazil
Ulsan Korea, South
Omsk Russia
Dakar Senegal
Abu Dhabi United Arab Emirates
Monterrey Mexico
Tripoli Libya
Rostov-on-Don Russia
T'bilisi Georgia
Fez Morocco
Birmingham United Kingdom
Yerevan Armenia
Cologne Germany
Tunis Tunisia
Islamabad Pakistan
Great, we could use the UN/LOCODE standards to determine site IDs and hostnames - two letters for country and three letters for city.
http://www.unece.org/cefact/locode/service/location
I don't see any offices in Ireland or Luxembourg so I suppose this enterprise is paying their taxes properly? :)
Quote from: SimonV on January 02, 2018, 02:00:29 PM
Great, we could use the UN/LOCODE standards to determine site IDs and hostnames - two letters for country and three letters for city.
http://www.unece.org/cefact/locode/service/location
I don't see any offices in Ireland or Luxembourg so I suppose this enterprise is paying their taxes properly? :)
What do you think the Libyan office is for? :smug:
For using the Nation/City combo, that's part of just about every company's naming standard. This would mean using our naming convention as a network access criteria is useless. Therefore, we're going to install certificates on everything. EVERYTHING. The Windows AD CA is up and running and ready to issue non-interactive certificates for all our devices.
No cert, no network access, that's the final state for all devices that can support a cert install. For those that cannot, we need to define ACLs that limit their connectivity to the rest of the network. We can't put everything behind a firewall, but we can put a lot of stuff behind an ACL.
Speaking of ACLs and country codes... IP address management... Yes, we're going into a full conversion of everything, but the database guys and developers are really digging in their heels in resisting switching their thousands of servers from using public addresses to an RFC 1918 address space. And don't even get them started about IPv6...
with all those certs, the CA private keys are really really important, will have to devise a way to protect the CA server, without hindering production, eg. daily cert generation.
GL with IPv4 in China... and probably all the Asia-pacific area.
Quote from: ristau5741 on January 11, 2018, 10:40:41 AM
with all those certs, the CA private keys are really really important, will have to devise a way to protect the CA server, without hindering production, eg. daily cert generation.
One of the best protections for the CA server is to have secondary servers set up that can do validation, then take the primary server off the network. Some orgs even go so far as to shut it down and to put it into a locked safe.
project funding must have dried up.
Quote from: ristau5741 on February 28, 2018, 09:58:38 AM
project funding must have dried up.
That happens, but good news is that we just got our Q1 budget approved. And some server guy has an idea about using NAT for the data center so that the servers keep their public IP addresses, but we do one-to-one static NAT to an RFC 1918 address... What's a nice way of saying :developers:
Aren't we using loadbalancers in the perimeter?
Just had a presentation on Viptella, and it looks pretty cool. I thought the piggybacking on the BFD packets for IPSLA was a neat trick. Would be a good choice for this project.
-Otanx
Quote from: SimonV on March 01, 2018, 12:46:36 PM
Aren't we using loadbalancers in the perimeter?
Yes, but management is concerned about not overloading the loadbalancers... so they put all they loadbalancers behind other loadbalancers. :wtf: