Current Mistakes Thread

[deanwebb]

root
password

I'm not going to say which box it is, but I have not yet changed that "easy password we'll use just for the demo..."

[that1guy15]

I recall at my last job, one of our guys threw in a debug ntp server and that crashed a 6500 VSS (IOS bug).
Thereafter, they decided that all debug commands required change management as well. sigh

Ouch! At a previous gig the MPLS team started deploying Juniper MX960s. Everyone ran around ranting and raving about how solid and reliable they were. Two days into production one of them crashed a full chassis just by connecting to the console... He said first the sup LED went red and then he just saw one by one each linecard lose power...

[routerdork]

I recall at my last job, one of our guys threw in a debug ntp server and that crashed a 6500 VSS (IOS bug).
Thereafter, they decided that all debug commands required change management as well. sigh

Ouch! At a previous gig the MPLS team started deploying Juniper MX960s. Everyone ran around ranting and raving about how solid and reliable they were. Two days into production one of them crashed a full chassis just by connecting to the console... He said first the sup LED went red and then he just saw one by one each linecard lose power...

I had the same issue with an ASR9K but luckily it just failed over to the other RSP.

[SimonV]

Yes, I tried it late last night. That fixed the policy sync issue, but this morning I had to move some VPN tunnels to new ST interfaces before they passed traffic again 

[SimonV]

Was adding another vsrx to my lab yesterday so I needed something to route between them. Found out the hard way that the CSR-1000V is throttled to 100kbps 

Any way to easily overcome this limitation? Settled for a VyOS appliance for now, which is actually pretty neat 

[LynK]

My last was trying to train a couple juniors how to add a FEX to a pair of 5Ks with vPC. I was lighting up a new FEX like I have done 1 milion times.

This fex was going to be 102 and I build all the config in notepad to lay out what was needed for them. When I pasted the config in I got an error on the Port-channel config. Nothing seemed wrong with the config on Port-Channel02 so I figured it was a one-off issue. I decided to remove the port channel and re-add it.

Code Select Expand

config t
no interface port-channel02


I then manually added Po102 and showed them how to a FEX is added. All went smooth

About 5 minutes later everything in the DC blew the f* up... If you already picked up on what I did props to you!

If not you will see that port-channel02 is not PO102. Thats a f'n "L". I just dropped PO2 not Po102...
Shit, shit,shit.. Po2 is the uplink between my 5Ks and 7Ks. I just dropped my whole DC!

Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.

Sometimes I hate this shit...

Can you please layout your exact conversation to your manager for our enjoyment... PLEASE!!!! 



My latest mistake was not changing the STP type on our new store switches. In these new switches, the default STP setting is edge. Guess what does work when you plug a device into it... a switch... aka multiple macs..

[deanwebb]

Truly, I did lol.

[NetworkGroover]

Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.

Sometimes I hate this shit...

Did you know that Arista switches support config sessions? 

Code Select Expand

BGPDC-SPINE2#configure session test

BGPDC-SPINE2(config-s-test)#int e4
BGPDC-SPINE2(config-s-test-if-Et4)#description test

BGPDC-SPINE2(config-s-test-if-Et4)#show session-config diffs
--- system:/running-config
+++ session:/test-session-config
@@ -25,6 +25,7 @@
    ip address 192.168.255.20/31
!
interface Ethernet4
+   description test
!
interface Ethernet5
!

BGPDC-SPINE2(config-s-test)#commit
BGPDC-SPINE2#sh run int e4
interface Ethernet4
   description test



[that1guy15]

Can you please layout your exact conversation to your manager for our enjoyment... PLEASE!!!! 

hehe, I got the team cake and ice cream the following week for causing a downtime!

We are in the middle of a very screwed up network and been cleaning it up for well over 2 years. Leadership is very aware we will step on a few land-mines as we move through this shit-pile. Yes this was a screw-up on my part and could have been avoided but it highlighted several flaws in our design so its a win for them. Plus the old team would have outages like this about 2-3 times a year and they would last well over 4 hours.

Conversation was pretty much me saying sorry I screwed up I fat-fingered a number. Here is what happened and here is how I fixed it. This is what we found and here is what I will do to make sure it dosent happen again.

[LynK]

 

lucky you.

[that1guy15]

Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.

I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...

[Nerm]

Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.

I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...

[LynK]

Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.

I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...

lmao

[warren.sullivan.526]

Years ago I connected a normal console cable to an APC UPS "serial" port, it forced a reboot, killed power to 7 48 port POE access switches......doh


Sent from my iPhone using Tapatalk

[deanwebb]

Years ago I connected a normal console cable to an APC UPS "serial" port, it forced a reboot, killed power to 7 48 port POE access switches......doh


Sent from my iPhone using Tapatalk

That is bad because it is not good.