Cisco router self-healing

Started by Dieselboy, September 21, 2016, 10:44:49 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions

Dieselboy

September 21, 2016, 10:44:49 PM
I brought up our new ISP circuit in Sri Lanka this morning. Yesterday I had it plugged in and I configured it remotely. Yesterday I established an additional IPSEC VTI tunnel across the new circuit - no issues.

This morning I couldn't get NAT to work on the new line. I set an additional route for 8.8.8.8 pointing out the new ISP and I could not ping 8.8.8.8 but the BOVPN was working fine.

Trying to edit the route-map for the NAT I literally entered "route-map NAT-MAP permit 200" and after pressing enter, all 3 of my VPNs dropped.
No amount of clearing crypto's or shut/no shut on the VTI interface resolved the VPN issues so as a last resort, to clear my config changes and to get it working for the SL users as they would be starting their day soon I done what I hate and rebooted the router. All VPNs worked again.

I decided to raise a TAC for the NAT not working. I took a tech-support and whilst filling out the new TAC form describing the network and what I've done / what I was trying to do I wrote that I could not get any NATs out the new ISP. I then went to the router and done a show ip nat translations, and grepped it for our new WAN IP. To my surprise I could see NATs.

So I went back to my RDP session to an inside host, started the ping again to 8.8.8.8 and the pings were getting replies. I could also see this ICMP being natted to the new ISP.

So, it seems that if you have an issue with an IOS router, take a tech-support and try and raise a case. The router knows this and fixes itself.

:zomgwtfbbq:

EOS

#1
September 22, 2016, 05:55:31 AM
LOL!!

Crazy how that works sometimes.

wintermute000

#2
September 22, 2016, 05:57:57 AM
In all seriousness, high chance the reboot did something

deanwebb

#3
September 22, 2016, 10:18:01 AM
It's a heisenerror. The act of observing the error caused it to disappear.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#4
September 22, 2016, 08:56:58 PM
Quote from: wintermute000 on September 22, 2016, 05:57:57 AM
In all seriousness, high chance the reboot did something

Yep - defo a bug but it's not in the bug toolkit for this ios. Didn't bother raising a tac as it's working now as I configured it the first time.

Quote from: deanwebb on September 22, 2016, 10:18:01 AM
It's a heisenerror. The act of observing the error caused it to disappear.

It's a quantum-error.  :awesome:

SimonV

#5
September 23, 2016, 02:55:01 AM
I had similar issues with NAT in the past. Didn't work while I was sure about the config. Entering it again made no difference but after a simple reboot it started working. But on an old 877 router so I would hope that IOS has improved since then...

Dieselboy

#6
September 23, 2016, 05:41:30 AM
Ever since January 2014 when I started this job, it seems like if there's a bug either discovered or yet undiscovered then our environment will find it :)

I can't remember but I think that the issue was still there after the reboot but the reason for the reboot was to revert everything and get it working for the user's who would be arriving
Print
Go Up Pages1
User actions