WAR STORIES!

[deanwebb]

I used to work at a place where I discovered the enterprise root bridge was an access switch in a random wiring closet.

 

[that1guy15]

My current network is a Hospital system. We have multiple independent businesses that reside on campus and have PCs and such scattered all over the place. The old admins found the best command to segregate their network from ours and still pass traffic, spanning-tree bpdu filter.

This command is all over the place in some of the most random places and not is places that it should be.

After I came on I did a core replace and about once a month the network would blip. IM would drop, phones would go offline. Internet would die for a second or two. Every time MST root was shifting away from the core. Once I figured this out I traced it to the closet and what did I find? A single 2950 of one of these companies with Spanning-tree vlan 1-4094 priority 0. My MST was 4096...

It was a chain reaction. The 2950 had two up-links to two separate closets. One had bpdu filter inplace and the other had it removed with the core upgrade. Their STP killed the non-bpdu filtered path normally but every once in a while the other link would fail and STP would converge and take over my whole campus root.

Not fun and I am almost done segregate these companies into their own space.

[LynK]

My current network is a Hospital system. We have multiple independent businesses that reside on campus and have PCs and such scattered all over the place. The old admins found the best command to segregate their network from ours and still pass traffic, spanning-tree bpdu filter.

This command is all over the place in some of the most random places and not is places that it should be.

After I came on I did a core replace and about once a month the network would blip. IM would drop, phones would go offline. Internet would die for a second or two. Every time MST root was shifting away from the core. Once I figured this out I traced it to the closet and what did I find? A single 2950 of one of these companies with Spanning-tree vlan 1-4094 priority 0. My MST was 4096...

It was a chain reaction. The 2950 had two up-links to two separate closets. One had bpdu filter inplace and the other had it removed with the core upgrade. Their STP killed the non-bpdu filtered path normally but every once in a while the other link would fail and STP would converge and take over my whole campus root.

Not fun and I am almost done segregate these companies into their own space.

wow....lol.

Its okay, when i first joined my new company we had 2-3 variants of spanning tree running. The nexus 7k was running at default priority 32xxx. /facepalm.

[Otanx]

I have mentioned this before, but when we were doing clean up we found that all three cores had spanning-tree turned off. The only thing that saved this network is that when you turn spanning-tree off the switch will forward the bpdu.

-Otanx

[that1guy15]

Damn! Ive never walked up on a network with STP disabled...

My current network has about 3-4 major design issues that I am reverse-engineering and correcting. Each with their own level of head-desk moments. Its landmine after landmine on this guy. I will say though its not the biggest and most glamorous network but the experience gained and learning is off the charts. I will wear the scars of this network proudly for a long time!!

[routerdork]

Damn! Ive never walked up on a network with STP disabled...

I have. We acquired a company that did this. Some of the guys we acquired were telling us about how the network would just randomly "blip" and then it would come back, but it's always happened so don't worry about it. Had the luxury of interviewing one of the Sr. Engineers a few months later. My boss made sure to ask STP questions. LOL the guy used the same network as an example during the interview, they turned it off to increase speed. Bahahaha 

Needless to say it was re-enabled and blips suddenly went away. Weird.

[deanwebb]

Man, I'm glad I did the Hitler Rants about STP being turned off video...

[that1guy15]

Sounds like this needs to be split off into a "war stories" thread. I still have my GRE/QnQ/no CDP/NAT/tunnel network to tell you guys about

[deanwebb]

Consider it done... War stories thread starting in 3... 2... 1...

[deanwebb]

INCOMING!

What are the best, the worst, and the weirdest networks you ever worked on?

One of the worst for me was the server/comm room that had a working sprinkler system... and then a hub overheated and started to generate smoke... 

[ChestHair]

I had a client that installed a mini-split air conditioning unit above a server rack in an 8 x 12 closet. One humid day and an improperly installed drain, and that thing started to dump water. I showed up on a Monday morning and my jaw dropped as I opened the door. The saturated drywall was peeling off the wall and had fallen onto the top of the rack. Water was dripping down both sides, etc. Nobody seemed to be half as concerned as I was.

[NetworkGroover]

I used to work at a place where I discovered the enterprise root bridge was an access switch in a random wiring closet.

 

Default priority - everywhere.

[deanwebb]

Intermittent wireless issue... we'd have wireless access, then it would go off suddenly, then it would come back on, something was wrong with the hardware... So I asked where the wireless access point was mounted and nobody knew.

So I went to the switch, traced it down... it was IN MY OFFICE!

So I go back to my office and look around and, behind my chair, is the access point. Turns out, every time I got comfy and leaned back in my chair, the wireless went out. Someone would call, I'd sit up to answer the call and, oh, wait, nevermind, the wireless is back on. I'd check email, get bored, lean back again and then RING!

I found the problem: The power cord to the AP was getting pulled out *_just_enough_* when I tilted back for it to unseat in the outlet. When I tilted back forward, it moved back just enough to get power. So I moved it to be on top of a bookcase and that ended the intermittent wireless, which had been going on ever since it was set up, it seems. We also had better reception with it there, so I was the hero of the day.

[Otanx]

Let me see...

1. 200+ Switch layer two domain using VTP. We never killed the network, but we were extra careful. STP root was default everywhere so root was an edge switch in a small building.
2. The one I mentioned above. Cores had STP turned off. All other switches were using default STP configurations. So some were using STP some RSTP. Depending on how old the switches OS was.
3. A "failover" configuration of GRE with two head-end routers terminating remote sites. Sounds like a good plan, and if the primary GRE router died the backup would take over. However, if the backup GRE router was rebooted it killed all traffic. We never did figure out why. We just decommissioned the entire setup.
4. While in the Army stationed in Korea. Someone took a rack out of a dish washer, you know the piece with all the plastic sticks to hold the plates, and mounted that to the wall. They then zip tied a switch into this, and used the dish rack as cable management. I used to have a photo of this, but I can't find it now.
5. Korea again. Used pair-gain DSL modems to get connectivity over copper cable installed during the Korean war. Cable ran under the runway, and could not be replaced. Whenever it rained we lost connectivity.
6. Korea again. During a field exercise we needed to link the primary operations center with the backup. Didn't have fiber. Ran an entire box of CAT5 (1000'). Put ends on it, and plugged it in. It worked, but the entire network ran at a crawl.
7. Working with the Marines. Needed a buried cable run about 400'. How do you dig a trench 400' long? You steal a fork lift use an expedited government requisition process to get a fork lift, remove one fork, and rotate the other fork so it is facing down. Jam the fork into the ground, and drive forward. It the fork lift gets stuck push it with a truck.
8. Rats chewing on an unprotected fiber run in a building? Line up about 50 glue rat traps along the path, and lay the fiber in the middle. Replace glue traps as needed.
9. Wireless signal was poor. The other group running the wireless finally asked for help. We go over in force (5 network engineers) to troubleshoot. Found they had zip tied the AP to the steel I-beam that held up the roof. This put a good half inch of metal between most of the building, and the antenna.

That is all for now.

-Otanx

[icecream-guy]

few jobs ago, worked for a company that spent 500K on some beefy servers, but didn't take into account heat dissipation, or the need for A/C. it's was 110-120 in the server room on a daily basis. the servers burned themselves out in about a year.

Mystery of the down switch. Last job, we did a closet upgrade, due to the small closets the switches were mounted the switches vertically.  ports up, due to short cables. every so often a switch would go down for unknown reasons,  turned out the power plugs were falling out of the back (bottom) of the switch.