Cisco is at it again!! (Stable releases... NOT)

LynK · May 26, 2015, 03:23:53 PM

well....

Since we all know cisco is known for its stable releases, I thought we should create a thread to show the funniest (a.k.a - worst) bugs that you have come across. Make sure you specify which version, as well as the bug ID. The second one is pretty bad... if you are on it... upgrade STAT!

Browsing through the new 2960x IOS release, I decided to go to the caveats, and I found this beauty:

Version: 15.2.3E1 (ED)
Bug ID: CSCuo55798
Headline: Priority Queue Latency increases significantly during congestion (LOOOOOL)

Version: 15.0.2-EX5(ED)
Bug ID: CSCur56395
Headline: SFP issues (link flap on 10G SFP interfaces)

deanwebb · May 26, 2015, 03:57:40 PM

Always be on the lookout for microcode updates: http://tekcert.com/blog/2012/04/07/upgrading-3750x-can-take-longer-you-think
Going from 12.2-53 to 12.2-58 can take an extra 30-45 minutes to finish.

Then there's the cute trick you have to do when upgrading from 12.2-58 to 15.0 on your 4500s... http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/release/note/OL_24829.html

QuoteIf a switch uses a config-register ending in 0x2, it may drop into ROMMON if the bootup is interrupted by a powercycle.

Workaround: Use config-register 0x2101. CSCue19458

We hit this issue the hard way, with our 4507s going into ROMMON mode when we upgraded them to 15.0. We were all like

and the switch was all like

which made us all like

even more until we found the above note and then we were all like

when our Cisco rep came by for a visit to see how the upgrade went.

wintermute000 · May 26, 2015, 08:29:36 PM

The old Cat4000s used to do this all time - go into ROMMON on a reboot - you just typed boot and it kept going

Dieselboy · May 29, 2015, 03:20:02 AM

O.M.G. you'll love my recent bug then...

I have a 2921 router, running as a SSL VPN server amongst other things, and we use remote IP phones (9971 and 8945, mainly) and these use AnyConnect VPN app on the phone to connect back to the office. We were running 15.3.3M3 and we were hitting a memory leak bug due to HTTPS / SSL VPN. This consumed all I/O pool memory as a symptom. The fix was to upgrade to 15.3.3M4, "but you wont believe what happened next"

The issue is, from all new IOS versions, the SSL VPN component in IOS expects a DTLS request header from the VPN client to negotiate SSL VPN on DTLS. Since there is absolutely no IP phone firmware whatsoever to send a response to this DTLS request, no AnyConnect phones at all can establish UDP SSL VPN, on newer IOS. The result is voice over TCP, constant phone reboot / lockup / call disconnection and poor audio quality when it does work. Pinging a remote VPN phone which is 150ms away results in a soon as the call is answered, latency shooting up to 1000ms and beyond then timeouts and zero audio.

Affected IOS versions are:
a) 15.3.3M4 onwards (All releases onward)
b) 15.5(1)T onwards (All releases onward)

BUG ID: CSCup56792 (Private / Internal only bug) - although I don't know if this is a bug for the issue I've mentioned but in fact an enhancement request to have this "feature" implemented.

Who in their right mind would implement a "feature" that breaks the entire SSL VPN side of the telephony handsets and not even having a planned firmware release to work with the new feature. I have a TAC case raised, titled "new IOS feature breaks AnyConnect phones" and there is not even a job tasked for the devs to implement this into the IP phones to support the IOS head end.
I've no idea if this feature has been implemented into ASAs...

wintermute000 · May 29, 2015, 03:53:15 AM

Wtf!!!!!

icecream-guy · May 29, 2015, 08:24:48 AM

we've still got a feature request to to be able to run ASDM on an ASA in multicontext mode that allows for two-factor authentication. not yet in the planning stage. Status unchanged for about a year now.

Otanx · May 29, 2015, 09:25:06 AM

Technically not a bug, but a field notice. Still my favorite.

http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html

Plugging a cable into port 1 may cause the switch to reboot, and wipe the start-up config.

-Otanx

NetworkGroover · May 29, 2015, 10:50:37 AM

Quote from: Otanx on May 29, 2015, 09:25:06 AM
Technically not a bug, but a field notice. Still my favorite.

http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html

Plugging a cable into port 1 may cause the switch to reboot, and wipe the start-up config.

-Otanx

Haha - that's hilarious!

deanwebb · May 29, 2015, 12:10:36 PM

I really enjoyed the laugh. Of course, if it happened to me...

Dieselboy · June 02, 2015, 09:16:14 PM

Quote from: Otanx on May 29, 2015, 09:25:06 AM
Technically not a bug, but a field notice. Still my favorite.

http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html

Plugging a cable into port 1 may cause the switch to reboot, and wipe the start-up config.

-Otanx

Love it!

icecream-guy · June 30, 2015, 07:18:14 AM

hit another interesting tid bit this morning.

upgraded out 9K distro switches from 6.1(2)I3(2) to 7.0(3)I1(2) to enable interface flow control which is available in the new release for an application that needs it. core 9k's still running 6.1(2)I3(2).

OSPF process broke, caused a summary route from the core not to propagate to the OSPF peer distribution switches. thus leaving an island of unhappy devices that had no where to route.

TAC case opened and copying show-tech files now. can't wait to see what they say.

deanwebb · June 30, 2015, 09:12:47 AM

Current Vegas odds on the outcome of that TAC call:

Upgrade the core switch: 6:5
Reboot the core switch: 5:2
Reboot the distro switches: 3:1
Downgrade the distro switches: 7:2

icecream-guy · June 30, 2015, 10:25:48 AM

oh, they were downgraded, had about 20 minutes to troubleshoot or revert before the window closed. rollback decision was made and everything works as expected

NetworkGroover · June 30, 2015, 12:11:19 PM

Quote from: ristau5741 on June 30, 2015, 10:25:48 AM
oh, they were downgraded, had about 20 minutes to troubleshoot or revert before the window closed. rollback decision was made and everything works as expected

Yuck.

LynK · June 30, 2015, 12:43:47 PM

@ristau,

How are you allowed to go ahead with upgrades during normal operational hours, (or even off hours). If I say I am going to upgrade our 7Ks, and there is no outage (ISSU <3). The whole company goes nuts. Honestly... when I upgrade my core it is normally a year or 2 before I upgrade to a new-old version that has been tried and true...

I don't know... call me a baby.. but my upper management would flip bricks.

Access/Distrib. A O K. no problems upgrading.... core... DIR gives me the