Tweaking routing metrics

[that1guy15]

that1guy15 mentions unstable WAN links... what makes a WAN link unstable?

I use that as a pretty broad term. Your internal 3 foot fiber connections or IDF-MDF connections are always more stable than any connection to your ISP and to the internet. You also have lower latency and packet loss over these links as well. WAN links can vary greatly in this regard but they are better than they used to be.

Also routes/prefixes on the internet are constantly being added/removed and BGP was designed to handle a large number of prefixes (600K+ IPv4 now) and constant churning without falling over from having to process this. That is why you see BGP with timers and convergence so slow compared to IGPs.

IGPs actively processes when something changes. BGP passively processes based off timer.

When you bring BGP in-house in a more stable environment then the longer timers are not needed anymore.

[NetworkGroover]

Ok, so we did a test with a customer where they built out the same BGP DC with us and Cisco.  Ixia traffic was run in both directions and we did a slew of single-failure convergence scenarios.  The worst amount of packet loss duration we experienced was 4.67 seconds and only in one direction (was 69 milliseconds in the other).  This was during a Leaf/ToR reboot.  The second worst performance was after no-shutting a Leaf/ToR port after it had been shut down (I didn't write down exactly what the port was.. I suspect a host or uplink port though it was too long ago to remember) - and that was only 1.15 seconds in one direction (4 ms in the other). 

Other than that, everything was pretty much ~ 50 ms or less - a lot of them zero.  So I don't see how playing with timers is going to help you there.

Here's a spine BGP config from that environment - all spines configured similarly:

Code Select Expand

router bgp 64700
   router-id 10.146.16.11
   bgp log-neighbor-changes
   maximum-paths 64 ecmp 64
   neighbor BD_eBGP_GROUP peer-group
   neighbor BD_eBGP_GROUP fall-over bfd
   neighbor BD_eBGP_GROUP maximum-routes 12000
   neighbor 10.146.0.65 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.65 remote-as 64702
   neighbor 10.146.0.67 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.67 remote-as 64702
   neighbor 10.146.0.69 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.69 remote-as 64703
   neighbor 10.146.0.71 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.71 remote-as 64703
   neighbor 10.146.0.73 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.73 remote-as 64704
   neighbor 10.146.0.75 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.75 remote-as 64704
   address-family ipv4
      neighbor BD_eBGP_GROUP activate
      network 10.146.16.11/32

And here's a Leaf/ToR BGP config from that environment - all Leafs are configured similarly:

Code Select Expand

router bgp 64702
   router-id 10.146.16.13
   bgp log-neighbor-changes
   maximum-paths 64 ecmp 64
   neighbor BD_eBGP_GROUP peer-group
   neighbor BD_eBGP_GROUP fall-over bfd
   neighbor BD_eBGP_GROUP maximum-routes 12000
   neighbor 10.146.0.64 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.64 remote-as 64700
   neighbor 10.146.0.76 peer-group BD_eBGP_GROUP
   neighbor 10.146.0.76 remote-as 64701
   neighbor 10.146.0.105 remote-as 64702
   neighbor 10.146.0.105 next-hop-self
   neighbor 10.146.0.105 fall-over bfd
   neighbor 10.146.0.105 maximum-routes 12000
   address-family ipv4
      neighbor 10.146.0.105 activate
      neighbor BD_eBGP_GROUP activate
      network 10.146.16.13/32
      network 10.146.32.128/26

Here's my short-hand list of tests we did:

• Shut 40G between ToR1 and Spine1 at ToR1
• no shut
• Shut 40G between Tor2 and Spine1 at ToR1
• no shut
• Shut ToR1 port 1
• no shut
• Shutdown peer-link port 1 on ToR1
• No shut
• Reboot Spine1
• Reboot ToR1
• Unplug 40G between ToR1 and Spine1 at ToR1
• Plug
• Unplug 40G between ToR2 and Spine1 at ToR1
• Plug
• Unplug ToR1 port 1
• Plug
• Unplug peer-link port 1 on ToR1
• Plug
• Unplug Spine1 sup 1
• Plug
• Unplug Spine1 PS1
• Plug
• Power off Spine1
• Power on
• Power off ToR1
• Power on

[that1guy15]

dont disagree with you when you have BFD and Dead Peer Detection. There are scenarios when its not an option so Im trying to outline why.

[NetworkGroover]

dont disagree with you when you have BFD and Dead Peer Detection. There are scenarios when its not an option so Im trying to outline why.

Ah k - sorry if I jumped the gun.

[that1guy15]

dont disagree with you when you have BFD and Dead Peer Detection. There are scenarios when its not an option so Im trying to outline why.

Ah k - sorry if I jumped the gun.

Nah all cool, we are kinda drilling down into a single aspect of the original question pretty far

[icecream-guy]

Tweaking route metrics when redistributing from one routing protocol is a popular endeavor to show preference of which routes are seen in the routing table. some routing protocols requires a network engineer to set a metric for redistribution.

One could also change administrative distance of a routing protocol for the routing table to have preference for one routing protocol over another.
   
Network engineers tweak the cost for OSPF routing for preferring faster links.  The default cost for any link over 100Mbps is currently 1.
So traffic going over a 10Gbps link would have the same cost as going over a 100Mbps link. Changing the cost metric allows support for sending data over the preferred faster links.

[NetworkGroover]

Ahhh yeah I got lazer-focused on TIMERS which I loathe toying with.  TIMERS isn't really metrics.

Yes, tweaking routing metrics is a common exercise for traffic engineering:

• I want this flow to go this way while this other flow goes a different way, but I still want both paths available for HA
• I want corp traffic to take the faster path, but my less-important R&D traffic to take this slower path as to not hinder corp traffic - but again I need both paths available to either traffic for HA.

etc... etc..

[deanwebb]

Does one generally tweak metrics in conjunction with QoS or are those kept separate concerns?

[icecream-guy]

Does one generally tweak metrics in conjunction with QoS or are those kept separate concerns?

one may tweak a metric using a PBR to have the VOIP traffic take a faster path to a destination to keep the latency lower.
but generally routing table's and interface QoS are ships in the night.