Main Menu

Bad 6509

Started by dlots, March 30, 2016, 08:22:09 AM

Previous topic - Next topic

dlots

We had a 6509 go bad over a reboot last Friday: the system just wouldn't come back up.  After lots of trouble shooting and several RMAs we found we had
1 bad chassis
2 bad sup 720s
1 bad 10gb fiber line card
1 bad 48 port GE card
Not everything in the chassis, but quite abit.  I have never seen that much stuff crap out... espeshally for a reboot.

Gonna try and get them to do the Cisco GOLD thingy next time (basically runs the POST test while the system is running to make sure it will come back up.

routerdork

That was the only thing I hated about our 7600's. A reboot could instantly change a maintenance window. Only had one or two modules at a time though. You sir might hold a record now.
"The thing about quotes on the internet is that you cannot confirm their validity." -Abraham Lincoln

deanwebb

:ckfacepalm:

Yeah, that's a tough set of crap to hit your fan.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

icecream-guy

....and I _just_ finished reading this about 5 mins ago

Is the Cisco 6500 Series invincible?
http://www.networkworld.com/article/3049220/network-switch/is-the-cisco-6500-series-invincible.html



LOL
:professorcat:

My Moral Fibers have been cut.

deanwebb

Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

dlots

Quote from: ristau5741 on March 30, 2016, 08:39:01 AM

Is the Cisco 6500 Series invincible?



There we go, now the answer is yes... yes it is

NetworkGroover

Quote from: dlots on March 30, 2016, 08:22:09 AM
We had a 6509 go bad over a reboot last Friday: the system just wouldn't come back up.  After lots of trouble shooting and several RMAs we found we had
1 bad chassis
2 bad sup 720s
1 bad 10gb fiber line card
1 bad 48 port GE card
Not everything in the chassis, but quite abit.  I have never seen that much stuff crap out... espeshally for a reboot.

Gonna try and get them to do the Cisco GOLD thingy next time (basically runs the POST test while the system is running to make sure it will come back up.

Holy crap!  :wtf:
Engineer by day, DJ by night, family first always

wintermute000

I've had 50% of the line cards fail on me before thanks to this little gem. The fault doesn't reveal itself until reboot, the faulty module can run for years without any noticeable symptoms.

http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63743.html

deanwebb

Quote from: wintermute000 on March 30, 2016, 04:54:49 PM
I've had 50% of the line cards fail on me before thanks to this little gem. The fault doesn't reveal itself until reboot, the faulty module can run for years without any noticeable symptoms.

http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63743.html

:kramer:

NEVER. EVER. UPGRADE.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

Interesting. How does the unit still go on functioning when the memory goes bad? Pretty clever.

wintermute000

Its some kind of electrical tolerance thingy so the blades run fine in normal operation with normal electrical supply specs, then when subjected to startup/reload voltages or amps (don't ask me I'm no sparkie!) it craps out.


I talked to a few guys from my former MSP after I got nailed by it and they told me that at one point they had a level 1 grunt doing almost nothing but replacement calls for this particular fault, reckon they had several come in every night for a year or two. Because the ROMMON output is so specific its easy to nail it down to this bug.

mlan

We got hit with this memory component bug pretty hard in our 28xx router fleet, and I'm afraid every component in our 6509 is going to suffer the same thing on the next reload.

Dieselboy

So it's not just specific to the 6509 either?

EOS

DDAAAMMNNN!!!!!

That is not what you'd expect of a simple reboot.

deanwebb

And now you know why there are several zillion Windows XP boxes around the world, running business-critical applications, with brilliantly-colored post-it notes slapped on them, bearing the stern warning, "DO NOT REBOOT!"

:ckfacepalm:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.