unlucky :( mem leak

Started by Dieselboy, August 02, 2016, 12:43:26 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions

Dieselboy

August 02, 2016, 12:43:26 AM
I am aware of a bug in my current IOS code to do with SSL VPN and a memory leak.
Last weekend, I planned a reload to clear a leak (Saturday). Up time was 4 months as we had a complete power down 4 months ago to install server room AC.

Got an alert this morning (Tuesday) that free mem is below my configured threshold. I've been watching free mem drop off all day :(

Don't think I'll make it to the weekend.

deanwebb

#1
August 02, 2016, 09:02:32 AM
I don't think you'll make it to Wednesday.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#2
August 02, 2016, 09:15:48 AM
After posting this I confirmed that the memory leak was not due to a previous bug as this IOS has resolved that bug - so this is a new memory leak to me.

I done an upgrade to 1 release up, went from T3 to T4 release with fingers crossed. I cannot go any newer because Cisco have implemented a fix / enhancement into SSL VPN but failed to think about SSL VPN phones. Consequently this fix breaks SSL VPN phones. Going to speak to my account manager tomorrow for options.

Dieselboy

#3
August 02, 2016, 11:35:36 PM
I sent a lengthy email, politely worded to our Cisco AM, and given 8 TAC cases I've raised over the past few years which resulted in the memory leaks etc, 6 CSC bugs filed and 1 Field Notice. The other case is open still as raised yesterday.

I have asked for one of three options:
1. An IP Phone firmware engineering special, that I can use for our remote VPN phones to support the "fix" in the IOS SSL VPN. I Think this is a reasonable request, I'm not aware of the scope required in Cisco's back end though.

2. An IOS engineering special that does not include the "fix" but allows me to upgrade the router to mitigate known vulnerabilities. -> I don't want to do this for a number of reasons, but it's an option.

3. A free ASA-X AnyConnect IP phone license so I can move the phones to the ASAs. This is my option 2. if option 1 is not achievable.

May move this thread to another section if it gets more techy.

Dieselboy

#4
August 04, 2016, 04:35:12 AM
It's happening again

deanwebb

#5
August 04, 2016, 11:50:37 AM
Love the graphic title... and I'm thinking that migrating to the ASA may be the solution the Cisco guys might favor.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

icecream-guy

#6
August 04, 2016, 01:41:32 PM
Quote from: deanwebb on August 04, 2016, 11:50:37 AM
Love the graphic title... and I'm thinking that migrating to the ASA may be the solution the Cisco guys might favor.

ASA code ain't too clean either. at least the 9.1,  not sure about the higher rev trains.
:professorcat:

My Moral Fibers have been cut.

Dieselboy

#7
August 04, 2016, 09:11:35 PM
I was monitoring this last night whilst clenching the buttocks. RAM use has levelled out with 85mb free. Will keep an eye on it, but panick over for the moment.

Graph title was extreme irony ;) I sent this to the tac engineer, hope she found it funny too. I did explain in the email that I made a very rough prediction though, in case my humour was lost in context / translation.
:awesome:

>:D

wintermute000

#8
August 05, 2016, 01:23:45 AM
It could be worse. I once had a 3750X (IOS 15, natch) whose RAM leak locked out the AAA process.... you could see a memory alloc error on the AAA process on the console if you tried to (and failed) SSH. Had to physically reboot, couldn't even get in via console.

Dieselboy

#9
August 05, 2016, 03:24:37 AM
That's bad. I had that happen to this device but I don't use AAA for SSH access. I use RADIUS for the SSL VPN authentication and I had the exact same issue - no one could log in to the VPN on this box with the same error log.

So in short - don't add complications to your management process ;)
Print
Go Up Pages1
User actions