Internet Edge Design Layout

[LynK]

Hey guys!

I think it would be interesting to see visios/diagrams of your environments internet edge. If you also want to include sanitized configurations that would be pretty interesting as well. It would be cool to see the different thought processes behind each design, and why you ultimately came up with your solution.

[deanwebb]

INTERNET -> Bulk router that kills tons of traffic -> Firewall -> IPS -> Proxy server -> End users

From end users outbound, same process except there is no default route outbound. That kills off botnet traffic. All web traffic has to cross the proxy, and that can kill off other bad stuff.

[LynK]

@dean,

No switches going from your internet to your router? No switch between router and firewall? 

How are you engineering inbound/outbound traffic? Are you using HSRP? Or peering directly from FW to Routers.

[Dieselboy]

In what context are you searching for information? You mention peering and hsrp - so, large enterprise design? I am too, interested in this. So I am keen to read feedback on it.

I run an SMB and at the moment I have:

100M Copper handoff > 1110 router > outside switches > ASA HA pair > NX 3k core switches (inside)
and
100M VDSL (backup) > VDSL router > outside switches (switch 2) > ASA pair /^

- Nexus 3k runs HSRP for lan subnets at the moment.
- Nexus 3k gateway is the primary ASA, with standby IP and MAC configured so during failover, re-ARP is not required.
- ASA gateway is the single 1110 router with a tracked route. 2nd default is the VDSL router.

- IPSEC VPNs are VTI tunnels
- IPSEC VTI Tunnel per-internet connection (eg. 2 tunnels for the main site to remote site, one for each connection)
- BGP runs across the VTI tunnels with lowered timers.

I would like to set up a resilient BGP peering but there are a number of blockers for this.


BTW - in some very small start ups / big financial company spinoffs, the internet edge has looked a lot like this::   

[deanwebb]

@dean,

No switches going from your internet to your router? No switch between router and firewall? 

How are you engineering inbound/outbound traffic? Are you using HSRP? Or peering directly from FW to Routers.

Switches do security? I don't understand your question... 

Lol, ask a security guy a question, get a security answer, even if it's a routing/switching question.

At any rate, I was describing a security edge that I've seen at quite a few enterprise customers and at former_employer.com. Pretty sure there were switches involved, couldn't tell you if they were doing HSRP or ID-10T. Now that I think about it, there's usually a WAN accelerator and a link redundancy appliance in there.

[deanwebb]

BTW - in some very small start ups / big financial company spinoffs, the internet edge has looked a lot like this::   

And you left out the unpatched Windows 2003 server running RDP, with no credentials required for access.

And in a branch office of a large organization, the unpatched Windows 2003 server running RDP with no credentials required for access has a NIC that connects it directly to the corporate network. Roughly 12 hours after the Byelorussian criminal gang launches the cryptolocker attack, I get called to see what I can do to help.

Problem is, most firms don't like it when you add "Byelorussian criminal gang" to their network diagram Visios.

[Nerm]

Neat idea....I'll play! Here is a sanitized HLD version of my world. Feel free to criticize as you see fit.

[LynK]

Dieselboy,

How do you handle failover when im assuming you have a different IP scheme per provider? Or do you have ARIN addressing? If not, that must be fun.

@Nerm,

So you have two ISPs, one at each site, how are you handling DNS & external IP failover if DC1 goes offline? I'm assuming you have two different IP spaces (one for each provider).

[LynK]

Here is my edge design.

Since our main DC is in a pretty rural area, we do not have options for multiple metro-e providers, however one of our branch sites does. So what we are designing is carrying that internet over our spine leaf, through a layer 2 vxlan vlan to DC 1, then to edge router 2. If we lose connections to the spine/leaf, no big deal. Only 1 ISP goes offline.

[Nerm]

We have almost no external facing services and what we do have is not business critical so we don't auto-failover that between DC's. If DC1 goes offline we manually change less than 10 public DNS records to point to DC2.

[icecream-guy]

We have almost no external facing services and what we do have is not business critical so we don't auto-failover that between DC's. If DC1 goes offline we manually change less than 10 public DNS records to point to DC2.

what is the TTL on those DNS records?  could take 24-36 hours for those changes to propagate to the far ends of the internet.

[Nerm]

5 minutes. The SLA on those services is 5 days so not business critical. The stuff for us that is business critical is either outbound traffic or inside east-west traffic.

[Dieselboy]

Dieselboy,

How do you handle failover when im assuming you have a different IP scheme per provider? Or do you have ARIN addressing? If not, that must be fun.

Failover is outbound internet access and BOVPN connectivity, only. We're an SMB with <50 users, a bit like a start-up.
There is no resilient inbound access, just have a /28 subnet on the primary and a /32 on the secondary. However, one idea I had was leverage Azure cloud resilience and connect privately from azure via resilient IPSEC VTI for some services. Our data download rate (upload from DC) is low, probably wouldnt have a charge for a few web apps. Another challenge that this would tackle is POP. As we have most users working from their homes or other places, I do get issues with some geographic users due to local ISP peering issues going between countries. This would need proper thought and planning and I've not looked into it at that low-level.

We have almost no external facing services and what we do have is not business critical so we don't auto-failover that between DC's. If DC1 goes offline we manually change less than 10 public DNS records to point to DC2.

Are those all CNAME records?

[Otanx]

Our edge is a pair of Arista 7280SRs. Each is connected to our DIA, and an IX. They advertise our /24s. We then have a bunch of routed links in to our different services. We also use a couple Quagga boxes to advertise drop prefixes to our edge. Stuff like RFC1918, and other ranges supplied by our cyber team. uRPF drops traffic before anything sees that traffic. Each service is behind a firewall, or pair of firewalls. NAT is done on the firewalls when needed. All doing OSPF with the edge to steer traffic. Currently the Aristas advertise a default down, and accept whatever the service advertises.

-Otanx