Internet Edge Design Considerations

[LynK]

Hey guys,

Browsing through some documentation on internet edge designs/etc, and I am curious how y'all are doing this. Lets assume 2 ISPs with BGP (and you want active/active with failover)

Here is what I have seen from different design guides:

1) Run iBGP on all equipment, and advertise prefixes (half of internet on one edge router, and half of internet on other edge router).

2) HSRP with static routing (which really is not true active/active)

3) GLBP with static routing (I guess this will work, but meh)

4) Redistribute iBGP into OSPF and then advertise routes in ospf

5) (anything else)?

What is your preferred way of internet edge design (assuming active/active), and why?!

[wintermute000]

BGP everything, OSPF for loopbacks.  There's no reason (in utopia) why your entire DC isn't built like this. SP large-scale design best practices FTW

But failing that then BGP to OSPF is pretty standard. Just don't forget to avoid redistribution feedback, and that unless you are running MPLS (obviously not) then every router in the iBGP path must have the iBGP routes and the correct next-hops. I've seen people resort to GRE tunnels for this (works, if messy and then dealing with frag/MTU). Then there's also the religious debate about next-hop-self vs redistributing the ISP transit subnet (CCDE says latter for faster convergence, FWIW).

I assume re: iBGP you are referring to your internal peerings, as it will always be eBGP to the ISP.....
I also assume that you have multiple /24s to go active/active as you're not doing full tables (if you were then this question would be moot)

every HSRP/GLBP/static route "solution" is an instant fail

[LynK]

Yes I was assuming eBGP to the providers. haha.

I agree, that anything static is a fail. Lets assume you are getting a full table from your ISPs. Do you then advertise two default routes to your iBGP peers? Or do you kill them with the full table?

edit: Why do you need multiple /24's for active active?

[wintermute000]

Sorry not full picture. Let me elaborate.
24 is smallest prefix you can advertise to the internet. So if you have two blocks you can preference them differently for inbound. And hence active active.
You could advertise same block to both ISP same attributes then pot luck which inbound path is better, you'll get asymmetry.
Full tables let's you take optimal link bgp wise outbound. Again asymmetry. Which is fine from a pure routing pov but messes with services which tend to be stateful
There's also pbr and other duct tape. LOL

[LynK]

I got where you are coming from, from an inbound perspective. But from an outbound perspective, what is the best way to "load share" outgoing traffic. Would this be to advertise two defaults to downstream iBGP peers, and then allow maximum-paths?

Or advertise through route maps a prefix range (0-127), (then anything else). One goes advertises out one router, and the rest out the other.

[icecream-guy]

Last place we used one ISP link for inbound, the other ISP link for outbound.  Later on, had to apply filtering to advertise subnet routes from either one when one of the pipes got full..

[LynK]

Yeah. I believe I have a few options because you could either:

1) Advertise two full tables

2) Advertise two partial tables, and two defaults from the providers (then use maximum paths + multi path relax)?

3)?

[icecream-guy]

Rule #1: Do Not make yourself a transit link between the two ISP's

[LynK]

Rule #2: Always use BFD

[deanwebb]

Rule #1: Do Not make yourself a transit link between the two ISP's

Is there a story to go with this? 

[dlots]

What I would probably do is

Give Core 1 (or whatever your 1st hop out of the edge is) a default static route to Edge 1, and a crappy static route to Edge 2.  (redistribute the static route)

Give Core 2 a default static route to Edge 2 and a crappy static route to Edge 1. (redistribute the static route)

Edge 1 and 2 have iBGP running between them, but don't advertise routes they learn from one another out to the ISPs (In theory your ISP should limit what routes they learn from you, but unless you want to pay your ISPs to move data between them I would advise against trusting them).  I would advise setting up your BGP metric so Edge 1 to use ISP 1 and Edge 2 to use ISP 2.  If you have a full BGP table if a route isn't being advertised from 1 ISP but it is for the other the traffic will go to the appropriate Edge and still make it out, if you don't have a full BGP table if one ISP goes down you still go out the other edge.

Now you just need to tweek your routing on the Distribution devices to prefer Core1 or Core2, in this way you can balance how much traffic is going out which ISP by moving your Distribution devices between cores. This lets you have more control especially if you are doing a full BGP table.  For example if you are at a collage the ISP with the better route to Netflix, hulu, and youtube will always be getting hit way harder than the one with the worse routes.

The main downside to this is if one of your cores go down one of your ISPs will get hammered.

[deanwebb]

To fix that last issue, have a condition on the proxy to block FB, YouTube, and NetFlix if it can't ping one core or the other.

[dlots]

That would work

Another option if you have a full BGP table is to have an IPSLA on the edge that monitors the core,  If a core goes down it triggers an EEM script that modifies your BGP metric so they take the best BGP route.  And another one that when the IPSLA comes back up it puts the BGP metric back.

[wintermute000]

re: outbound, I'd probably prefer to (not knowing your specific requirements) Keep It Simple Stupid as follows, if you are open to running full tables
- Full tables + iBGP in edge block
- ECMP into edge block
- let full tables take it wherever.

PBR is also another option, although rapidly gets messy.

If partial tables or straight default, what dlot says - each router prefers a different ISP then you can control which link yourself in your internal routing. However most of the time, for enterprise at least, outbound traffic is much less of a concern than inbound balancing.

If active/active is not mandatory, classic design using AS-PATH prepend out the secondary link and inbound local-pref - maximum stability and simplicity, also keeps jitter to minimum (deterministic).

In any event you will need the asymmetric routing in front of any stateful services (FWs) so you won't be able to terminate ISP directly on firewalls using any non deterministic design.

re: not becoming transit, regex is your friend (the classic ^$)

Whenever customers ask for active-active I always ask them 'so if your normal demands are so high that you need to utilise your 'secondary' link as well, do you think performance will be acceptable if you lose your primary link?'.

[deanwebb]

Whenever customers ask for active-active I always ask them 'so if your normal demands are so high that you need to utilise your 'secondary' link as well, do you think performance will be acceptable if you lose your primary link?'.

... and then there was silence...



People shooting for five nines and max ROI simply do not understand the concept of "backup capacity". These are the same types that will insist that all business functions are 100% critical, everything is top priority, and that you can buy unicorns in bulk via eBay and Amazon Prime.

Once upon a time, 80% utilization of an asset was considered "full capacity". Anything over that was "over capacity" and would get the business owner to look at expansion plans before utilization got critical at 85%+.