Just realized I sent the wrong IP address for an A record in DNS.
Which is easier, changing the record, or switching functions between the box that *should* have had its IP address submitted and its mate?
In this company, the switch is easier. :wall:
Quote from: deanwebb on May 19, 2015, 09:46:51 AMIn this company, the switch is easier. :wall:
Surprisingly enough that's the case in many companies I've seen. One would say a decent DNS setup and procedure wouldn't be expensive, nor complicated.
The person in charge of external DNS is a marketer in charge of brand identity.
We had to explain the difference between a host and a subdomain to this person. The one in *charge* of DNS.
:zomgwtfbbq:
Quote from: deanwebb on May 19, 2015, 12:45:40 PM
We had to explain the difference between a host and a subdomain to this person. The one in *charge* of DNS.
I couldn't find a good facepalm emoticon so this will have to do here. :developers:
The summary of the email conversation:
Network: We would like an A record for our new host, please.
DNS/Marketing: OK, we need some more information before we create your subdomain.
Network: No, we would like a host. We do not want a subdomain.
DNS/Marketing: As soon as we get that information, we can get to work on the subdomain request. (In direct reply to the message above.)
Network: No. We want a host. Just a host. Just one A record.
DNS/Marketing: Ohhhhh, wait a minute... you want a host!
Network: Yes, that is what we want.
DNS/Marketing: So should we cancel the ticket for the subdomain creation, or are you still planning to submit the paperwork for that?
:wall:
Quote from: deanwebb on May 19, 2015, 03:43:20 PM
The summary of the email conversation:
Network: We would like an A record for our new host, please.
DNS/Marketing: OK, we need some more information before we create your subdomain.
Network: No, we would like a host. We do not want a subdomain.
DNS/Marketing: As soon as we get that information, we can get to work on the subdomain request. (In direct reply to the message above.)
Network: No. We want a host. Just a host. Just one A record.
DNS/Marketing: Ohhhhh, wait a minute... you want a host!
Network: Yes, that is what we want.
DNS/Marketing: So should we cancel the ticket for the subdomain creation, or are you still planning to submit the paperwork for that?
This is basically every conversation of mine with any other company providing a service or support that I have ever dealt with. I have to hide sharp opjects so I don't stab myself in the eyes.
Configured 7 MGCP Voice gateways and 8 3560s that will be used at a customer's remote sites. Forgot to add DHCP Pools for the phones to the first two routers. They are on the bottom of the stack. Easy to fix but I've annoyed myself.
Not my mistake but anyway :whistle:
[edit security]
'policies'
Policy is out of sync between RE and PFEs: cluster1.node1, cluster1.node0. Please resync before commit.
error: configuration check-out failed
{primary:node0}[edit]
Recommended solution: http://www.juniper.net/documentation/en_US/junos12.1/topics/task/operational/security-policy-security-device-synchronizing.html
:wall:
:doh:
Quote from: SimonV on July 22, 2015, 06:35:04 AM
Not my mistake but anyway :whistle:
[edit security]
'policies'
Policy is out of sync between RE and PFEs: cluster1.node1, cluster1.node0. Please resync before commit.
error: configuration check-out failed
{primary:node0}[edit]
Recommended solution: http://www.juniper.net/documentation/en_US/junos12.1/topics/task/operational/security-policy-security-device-synchronizing.html (http://www.juniper.net/documentation/en_US/junos12.1/topics/task/operational/security-policy-security-device-synchronizing.html)
:wall:
(https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRujZgi517gBYEZR-Btl-ddYX318WmobtkkvACO9syYT2kgC2lAaQ)
My last was trying to train a couple juniors how to add a FEX to a pair of 5Ks with vPC. I was lighting up a new FEX like I have done 1 milion times.
This fex was going to be 102 and I build all the config in notepad to lay out what was needed for them. When I pasted the config in I got an error on the Port-channel config. Nothing seemed wrong with the config on Port-Channel02 so I figured it was a one-off issue. I decided to remove the port channel and re-add it.
config t
no interface port-channel02
I then manually added Po102 and showed them how to a FEX is added. All went smooth
About 5 minutes later everything in the DC blew the f* up... If you already picked up on what I did props to you!
If not you will see that port-channel02 is not PO102. Thats a f'n "L". I just dropped PO2 not Po102...
Shit, shit,shit.. Po2 is the uplink between my 5Ks and 7Ks. I just dropped my whole DC!
Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.
Sometimes I hate this shit...
Funny you post that today. I did something very similar late last week. Was setting up ports for a server migration from our old 3560s to our new FEX setup. Copied the port configs on the 3560, and modified them to the Nexus syntax. Forgot to validate the Po numbers were not in use on the 5Ks. Pasted the config and everything seemed OK. Get a call from a systems guy. One of their servers isn't responding. Took me about 10 minutes to figure out that one of the Port Channels was in use, and when I pasted in the config I moved it to a new vlan. Oops.
-Otanx
I was going through our ISE policies and saw a policy rule labeled "TEST" with object elements also containing the word "test." I think, "Hmmm, this pesky test rule has been here forever and I'm tired of looking at it. I'm going to disable it!"...
Several days later I get a ticket that certain users have been having wireless issues for several days. Sure enough, that "test" rule was serving a production function. It pissed me off pretty badly. I'm OK making my own mistakes, but when someone drops a banana peel for me to slip on.....that's not cool.
Quote from: that1guy15 on July 22, 2015, 07:44:31 PM
My last was trying to train a couple juniors how to add a FEX to a pair of 5Ks with vPC. I was lighting up a new FEX like I have done 1 milion times.
This fex was going to be 102 and I build all the config in notepad to lay out what was needed for them. When I pasted the config in I got an error on the Port-channel config. Nothing seemed wrong with the config on Port-Channel02 so I figured it was a one-off issue. I decided to remove the port channel and re-add it.
config t
no interface port-channel02
I then manually added Po102 and showed them how to a FEX is added. All went smooth
About 5 minutes later everything in the DC blew the f* up... If you already picked up on what I did props to you!
If not you will see that port-channel02 is not PO102. Thats a f'n "L". I just dropped PO2 not Po102...
Shit, shit,shit.. Po2 is the uplink between my 5Ks and 7Ks. I just dropped my whole DC!
Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.
Sometimes I hate this shit...
I am not envying what you will have to go through re: PIR and other political knock on effects.
It happens to the best of us sometimes!
I recall at my last job, one of our guys threw in a debug ntp server and that crashed a 6500 VSS (IOS bug).
Thereafter, they decided that all debug commands required change management as well. sigh
root
password
I'm not going to say which box it is, but I have not yet changed that "easy password we'll use just for the demo..."
:naughty:
Quote from: wintermute000 on July 22, 2015, 10:37:29 PM
I recall at my last job, one of our guys threw in a debug ntp server and that crashed a 6500 VSS (IOS bug).
Thereafter, they decided that all debug commands required change management as well. sigh
Ouch! At a previous gig the MPLS team started deploying Juniper MX960s. Everyone ran around ranting and raving about how solid and reliable they were. Two days into production one of them crashed a full chassis just by connecting to the console... He said first the sup LED went red and then he just saw one by one each linecard lose power...
Quote from: that1guy15 on July 23, 2015, 07:58:22 AM
Quote from: wintermute000 on July 22, 2015, 10:37:29 PM
I recall at my last job, one of our guys threw in a debug ntp server and that crashed a 6500 VSS (IOS bug).
Thereafter, they decided that all debug commands required change management as well. sigh
Ouch! At a previous gig the MPLS team started deploying Juniper MX960s. Everyone ran around ranting and raving about how solid and reliable they were. Two days into production one of them crashed a full chassis just by connecting to the console... He said first the sup LED went red and then he just saw one by one each linecard lose power...
I had the same issue with an ASR9K but luckily it just failed over to the other RSP.
Quote from: wintermute000 on July 22, 2015, 07:17:29 PM
(https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRujZgi517gBYEZR-Btl-ddYX318WmobtkkvACO9syYT2kgC2lAaQ)
Yes, I tried it late last night. That fixed the policy sync issue, but this morning I had to move some VPN tunnels to new ST interfaces before they passed traffic again :whistle:
Was adding another vsrx to my lab yesterday so I needed something to route between them. Found out the hard way that the CSR-1000V is throttled to 100kbps :mrgreen:
Any way to easily overcome this limitation? Settled for a VyOS appliance for now, which is actually pretty neat :dance:
Quote from: that1guy15 on July 22, 2015, 07:44:31 PM
My last was trying to train a couple juniors how to add a FEX to a pair of 5Ks with vPC. I was lighting up a new FEX like I have done 1 milion times.
This fex was going to be 102 and I build all the config in notepad to lay out what was needed for them. When I pasted the config in I got an error on the Port-channel config. Nothing seemed wrong with the config on Port-Channel02 so I figured it was a one-off issue. I decided to remove the port channel and re-add it.
config t
no interface port-channel02
I then manually added Po102 and showed them how to a FEX is added. All went smooth
About 5 minutes later everything in the DC blew the f* up... If you already picked up on what I did props to you!
If not you will see that port-channel02 is not PO102. Thats a f'n "L". I just dropped PO2 not Po102...
Shit, shit,shit.. Po2 is the uplink between my 5Ks and 7Ks. I just dropped my whole DC!
Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.
Sometimes I hate this shit...
Can you please layout your exact conversation to your manager for our enjoyment... PLEASE!!!! :mrgreen: :mrgreen: :twisted: :pub: :cheers:
My latest mistake was not changing the STP type on our new store switches. In these new switches, the default STP setting is edge. Guess what does work when you plug a device into it... a switch... aka multiple macs.. :developers: :developers: :developers: :developers: :developers:
:coolstory:
Truly, I did lol. :)
Quote from: that1guy15 on July 22, 2015, 07:44:31 PM
Took me 5 minutes to figure it out and correct but my apps team spent the next 4 hours stabilizing their shit.
Sometimes I hate this shit...
Did you know that Arista switches support config sessions? :problem?:
BGPDC-SPINE2#configure session test
BGPDC-SPINE2(config-s-test)#int e4
BGPDC-SPINE2(config-s-test-if-Et4)#description test
BGPDC-SPINE2(config-s-test-if-Et4)#show session-config diffs
--- system:/running-config
+++ session:/test-session-config
@@ -25,6 +25,7 @@
ip address 192.168.255.20/31
!
interface Ethernet4
+ description test
!
interface Ethernet5
!
BGPDC-SPINE2(config-s-test)#commit
BGPDC-SPINE2#sh run int e4
interface Ethernet4
description test
Quote from: LynK on September 10, 2015, 03:27:14 PM
Can you please layout your exact conversation to your manager for our enjoyment... PLEASE!!!! :mrgreen: :mrgreen: :twisted: :pub: :cheers:
hehe, I got the team cake and ice cream the following week for causing a downtime!
We are in the middle of a very screwed up network and been cleaning it up for well over 2 years. Leadership is very aware we will step on a few land-mines as we move through this shit-pile. Yes this was a screw-up on my part and could have been avoided but it highlighted several flaws in our design so its a win for them. Plus the old team would have outages like this about 2-3 times a year and they would last well over 4 hours.
Conversation was pretty much me saying sorry I screwed up I fat-fingered a number. Here is what happened and here is how I fixed it. This is what we found and here is what I will do to make sure it dosent happen again.
:zomgwtfbbq:
lucky you.
Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.
I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...
Quote from: that1guy15 on September 15, 2015, 02:21:05 PM
Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.
I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...
:haha3:
Quote from: that1guy15 on September 15, 2015, 02:21:05 PM
Oh dont take that as my job is all sunshine and fluffy kittens. This network is a bitch, and will shank you in the side the second you take your eyes off it.
I cant wait for it to burn down so I can build it right or hand it off to the next sucker to run it...
lmao :glitch:
Years ago I connected a normal console cable to an APC UPS "serial" port, it forced a reboot, killed power to 7 48 port POE access switches......doh
Sent from my iPhone using Tapatalk
Quote from: warren.sullivan.526 on September 27, 2015, 07:53:09 AM
Years ago I connected a normal console cable to an APC UPS "serial" port, it forced a reboot, killed power to 7 48 port POE access switches......doh
Sent from my iPhone using Tapatalk
:zomgwtfbbq:
That is bad because it is not good.
Loading up the IPS user interface...
It asks to upgrade or to proceed without upgrading.
If I upgrade, I will hose up the IPS user interfaces of everyone in the company that works with the IPS devices. I do not want to click upgrade.
On my way to clicking "Proceed without upgrading", I twitch and click on "upgrade".
:rage:
Oh, is that my manager on IM? I wonder what's on his mind right now?
:printer:
:wtf:
Oh damn... sorry bro. :(
"And children this is why we don't do crack before we go to work" lol.
I have done that before myself where you go to click on one button and then sneeze or twitch and "oops" lol.
Quote from: Nerm on November 12, 2015, 12:20:45 PM
"And children this is why we don't do crack before we go to work" lol.
I have done that before myself where you go to click on one button and then sneeze or twitch and "oops" lol.
You know what that's going to lead to right? A new standard requiring confirmation windows have the OK/NOT OK buttons spaced at least 10 inches away from each other. ;P
You know... I used to look up to you guys.... but if getting on your level will result in involuntary body spasms :excited: ... I think I'll be ok with being dumb.
:problem?: