Messages

I thought I would post an update to my thread. 

We found a bug in the Cisco software running the GetVPN key-server routers - Cisco bug ID CSCuq18492.  They would lose connection to each other (despite being able to ping back and forth with no problems) every so often.  This caused an outage when both of the key-server routers went primary just before a re-key.  Some of the GetVPN group member routers were associated to the real primary and others associated to the new primary.  When the re-key happened both routers sent out different keys so when the group members received their new keys and began to use them traffic flows stopped but since EIGRP was excluded from encryption all of the new messed up links continued to appear to the routers as valid paths so traffic kept flowing down.  How did I find this out?  By a major WAN outage causing impact to the largest sites on my network.

After the code issue was resolved by upgrading to a flavor of 15.4 I then implemented some testing with a new design.  Specifically, putting EIGRP into the encryption so if there were to be a problem with encryption the EIGRP traffic would also stop flowing correctly, any affected EIGRP adjacency would drop, and the path would no longer appear as valid for traffic.

This also brought up another issue... if there is no EIGRP adjacency then how will a remote site router know how to talk to the GetVPN key-server router?  Floating static routes fixed this.  I'm using a static route with an AD of 220 so when EIGRP comes up the lower AD value will edge out the static route and traffic will flow through normal paths established per EIGRP.

This has been working now for a couple weeks back into production after tons of testing with application owners and allowing smaller sites to burn in for a while to make sure.  In fact, our provider had a cut fiber a week or so ago which dropped one of our GetVPN hub sites entirely.  The remote site that was up at the time in testing/validation simply dropped that single EIGRP adjacency while continuing to stay connected to the other two hub sites with zero dropped traffic and no reports of issues.

I figured I'd post this here since I couldn't find any other information on this anywhere else on the Internet in my searching.  I'm not saying it doesn't exist... I just couldn't find it.

TIL that you shouldn't edit/remove the access-list attached to a route-map applied to a 20GB connection between your data center and the rest of the enterprise without first removing the route-map policy from the interface.

sgtcasey,  9 stacks can't be wired that way as the stack cable is too short.  the extra lond stack cable would be needed

Yep.  We use the longer stacking cable for all switch stacks larger than 3 and still go 1 to 2, etc.

I've never seen it done that way before.  I always go 1 to 2, 1 to 2, 1 to 2 all the way down the stack with the last switch in the stack going to the top switch in the stack to complete the ring.  I'm not saying the way you have it won't work perfectly fine. 

Odd problem for sure.  It might be time for a TAC case or a good look through the Bug Search Tool.

Juniper lets you configure as many IPv4 addresses as you want on an interface

ip address <ip address> <mask> secondary

This does the same thing for a Cisco VLAN interface.  I've seen this way too many times in poorly implemented networks.

That fact you were seeing these errors on the stack ports makes me wonder if there might not be a busted stack port cable or some sort of weird software bug.  Are you doing any odd configuration with your stack ports?  How are the stack ports connected up?

Okay, so I'm at my work laptop and decided to fire up jperf and see if you are even able to set the UDP bandwidth value to xx.xG and the answer is no.  However, you can set the MBps to get where you need.  Someone correct my math if it's wrong.

10Gbps is 1250MBps

iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 1250.0M -n 1000000000 -T 1

However, keep in mind that if you're using a laptop or PC with a 1Gbps NIC you wouldn't be able to push 10Gbps anyway.  In fact, if you're running a Windows OS jerpf/iperf can rarely even get up to the full speed of your NIC anyway.  I'm not sure if a Linux OS based machine can.  The links I use jperf/iperf on are our WAN links which are much smaller in sized than the 10Gbps+ you'll find in a data center.

I see your test seems to be completing very quickly.  If you want to run it longer just increase the number of 0's in the -n value.  That is the amount of data to transfer.  I usually set that number to 1,000,000,000 which is 1GB of data.  That way it takes 30-120 seconds for my test to run depending on the site and the pipe size.  That's enough time to get some good data.

I suppose if you set up several jperf/iperf servers/clients you could all get them to run at the same time to try and max out that 10Gbps link. 

...#YOLO

Not on my network... we've got patients to think about. 

Using UDP might give you a better idea of the total throughput with anything getting "in the way".  But increase the amount of UDP traffic you're trying to send.

iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 200.0M -n 1000000000 -T 1

The above is only going to try to send ~200Mbps.  Bump that up to something higher than 10Gbps since the link you have is running at 10Gbps.  I use jperf to set up the testing and iperf as the server on the remote machine so I'm not 100% of the command line but see if this might work:

iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 15.0G -n 1000000000 -T 1

An example of where I use this is today I wanted to test total amount of traffic I could get through a new 150Mbps link that was just turned over to my team.  I did the usual TCP testing to see throughput with QoS and crypto enabled and such.  Then I disable QoS and crypto and just spammed UDP traffic across it at 300Mbps (I usually double the bandwidth the link is set at by the provider) to see how much made it through.  I was getting right around 189Mbps across it.

Something to keep an eye out for with UDLD is if anything at all stops the UDLD hello packets (or whatever they're called) your fiber links will drop.  If you do not have errdisable recovery cause udld enabled along with errdisable recovery <time> (think those are the commands) then you'll need to go to the switch stack and console into it to bring it back online.

I've always kept UDLD going on device using fiber to prevent a bad fiber pair from throwing problems into the network.  Drop the busted fiber, I get the alert, send a ticket to the cable folks to test/replace as needed, bring back online once fixed.  Easy.

Until one fine morning around 2:00am my pager goes off that most of the access switches at a site (a 3 hour drive away) just dropped offline.  I go to connect to the core 4507 to check it out and nothing.  An hour or so later (thankfully I was able to reach a local contact who I talked through consoling into stuff for me) every thing was back up.  The cause was a bug with our 4507 code where the supervisor would simply stop processing/forwarding traffic.  It simply stopped working but from the outside it looked fine (all lights green and such).

When it stopped sending traffic that included all UDLD stuff.  All the access switches doing what they were configured to do err-disabled all of the fiber links due to UDLD requiring someone to physically go to them all to bring back online.  We decided to add in the errdisable recovery udld command so the ports would come back up automatically if this event happens again.  If we have a true UDLD issue the port will bounce and alert the team and someone can just go shut it manually.

So far so good.

We've been seeing this with 2960S and 3750X stacks for a couple years now.  This issue is one of the reasons we've decided where I work to not stack 3750X's more than 5.  Even at 5 sometimes they're just plain sluggish on the command line.

Said 3845 with the 7 years of uptime is now sleeping peacefully on the floor of a network closet.  The next step is to power it up, wipe the config, update the asset management application, and then decide if we keep or recycle.

Like most, I've done it.  Sitting there thinking, "I don't need a change to add a simple VLAN to this trunk!"

Type - type - type - ENTER

Wow, it usually doesn't sit there this long "thinking".  Oh shi...

* sgtcasey grabs laptop and console cable and runs to the data center to bring the server access switch back up.

[iperf -c 172.1.1.10 -p 5001 -t 15 -i 1 -f m -w 50MB]

May I please know what command you ran for your iPerf? I know the command can give you false results if you write it wrong.  So I want to be sure that's correct.  This is what I'm running iperf -c 172.1.1.10 -p 5001 -t 15 -i 1 -f m -w 50MB

Sure.  I'll set up a laptop connected to the remote router I want to test WAN bandwidth over.  On that laptop I start up iperf -s (for TCP) or iperf -s -u (for UDP).  Then on my work-issued machine I'll go to the other end of that WAN link and depending on what I want to use I'll run the following.

TCP - iperf.exe -c <iperf server IP> -P 1 -i 1 -p 5001 -f m -n 1000000000
UDP - iperf.exe -c <iperf server IP> -u -P 2 -i 1 -p 5001 -f m -b 200.0M -n 1000000000 -T 1

I use TCP to see what kind of performance a normal TCP session will get over a link.  For UDP I tell it to just spam tons of traffic through the WAN link.  I then check the other side interface stats to see what is actually making it through.  You do need to change how your iperf server is set up depending on if you want to use UDP or TCP.

On-call this weekend so not much.  I need to sit down and do some reading on QoS, though.  I either don't understand what I'm seeing happen or I've got a bug on an ASR-1006.