WAR STORIES!

Started by deanwebb, March 09, 2015, 02:49:00 PM

Previous topic - Next topic
Print
Go Down Pages 1 ... 3 4 5
User actions

Otanx

#60
February 09, 2017, 10:32:01 AM
Quote from: Hunterman1043 on February 08, 2017, 08:21:54 AM
Quote from: Otanx on February 07, 2017, 12:36:34 PM
typical time frame for a company to detect a breach is over 200 days.

That's comforting.

Download the Verizon Data Breach Report. It requires a free registration to download. It covers this stuff and is very well written.

-Otanx

ggnfs000

#61
March 20, 2017, 08:06:55 PM
Quote from: that1guy15 on March 09, 2015, 04:45:26 PM
My current network is a Hospital system. We have multiple independent businesses that reside on campus and have PCs and such scattered all over the place. The old admins found the best command to segregate their network from ours and still pass traffic, spanning-tree bpdu filter.

This command is all over the place in some of the most random places and not is places that it should be.

After I came on I did a core replace and about once a month the network would blip. IM would drop, phones would go offline. Internet would die for a second or two. Every time MST root was shifting away from the core. Once I figured this out I traced it to the closet and what did I find? A single 2950 of one of these companies with Spanning-tree vlan 1-4094 priority 0. My MST was 4096...

It was a chain reaction. The 2950 had two up-links to two separate closets. One had bpdu filter inplace and the other had it removed with the core upgrade. Their STP killed the non-bpdu filtered path normally but every once in a while the other link would fail and STP would converge and take over my whole campus root.

Not fun and I am almost done segregate these companies into their own space.

Wow, I will pretend that I understood anything.

SimonV

#62
September 20, 2017, 05:39:45 AM Last Edit: September 20, 2017, 05:49:04 AM by SimonV

deanwebb

#63
September 20, 2017, 09:55:42 AM
At the very least... but, proper connections to/from China should be via properly terminated WAN links...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#64
October 27, 2017, 08:43:23 AM
A parenting war story...

My daughter, 15, is serious about doing computer animation. It's a real passion for her, and that's a good thing in my mind. So, as a dad, I want to help out.

Well, I'm not using my lab *all* the time, so I ask her how long her last rendering project took. She says it ran for a day, got only to 10%, and then she canceled it and started clicking through specs for beefier boxes as she dreamed of spare CPU cycles.

I tell her that I've got an 8-core box with 64GB RAM, how about I spin up a VM for her to do the processing?

Her response: OH HELL YES

Gave her a TB of hard drive space, what the hell. Took about 30 minutes to get it all going and to teach her about how to RDP to another box.

She thanked me, went back to finish her school and then started copying files to the server after she was done with classes.

Around 6:18 PM, I hear my server go whhhhhrrrrrrrrrRRRRRRRRRRRRHEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

Ah, she's started her rendering!

3 hours later, she was at 100% and I was out in front for "Dad of the Year" honors. The server is back to 3% overall CPU utilization, but I know it'll get fired up again when she needs to render again in our family private cloud.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Nerm

#65
August 17, 2018, 06:24:31 PM
So today a colleague from a sister company learned why VTP can be dangerous. Especially when misconfigured or not configured at all and left in the default state.

deanwebb

#66
August 18, 2018, 01:16:58 PM
Quote from: Nerm on August 17, 2018, 06:24:31 PM
So today a colleague from a sister company learned why VTP can be dangerous. Especially when misconfigured or not configured at all and left in the default state.

:drama:

Go on...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Nerm

#67
August 18, 2018, 08:59:44 PM Last Edit: August 18, 2018, 09:01:25 PM by Nerm
They were migrating some legacy systems from an old environment to a new one. They decided to accomplish this by spanning some VLAN's via an L2 cross-connect between the environments. They configured it all up and brought up the cross-connect interfaces. As soon as they did the new environment they were migrating to went completely dark.

:whatudo:

I got called in to help sort it out as this environment was running some extremely important production systems. We console into the core switch for the new environment and immediately they were like "what are those VLAN's and where did they come from". Since I already mentioned VTP you can already imagine where this is going.

:bole:

The core switch in the new environment had been left mostly in default config. Basically unboxed it added some VLAN's and IP's and built a production environment on top of it. This meant VTP was left in its default "empty" config. The old environment they were migrating from was actively using VTP. When they brought up the cross-connect VTP did what it was supposed to do and wiped the VLAN's from the new environments core switch and injected all the VLAN's from the legacy environment.

:explosion1:

I got to be the hero and save the day, but the worst part is I had been consulted on this project a while back and I suggested the cross-connect be L3. They had picked L2 because it was "less work".

:facepalm2:

icecream-guy

#68
October 23, 2018, 11:45:02 AM
Got paged middle of the day, yesterday,
customer couldn't communicate between policy servers,  say 172.16.142.x to 172.16.143.x
Important first question, is this a new service?  (new services are not considered emergencies), customer says no
so hour or so troubleshooting, firwall looks find,  don't see anything in my capture,  so I engage network engineer
so he's troubleshooting for like and hours, and asks the second most important question, how long has this service been running. customer says since Saturday
Third most important question, has it ever worked?  customer says yes.  so routing and switching all looks good  arp tables populated, CAM too. all looks good.
so we start with the end user confoguration (which we souldn't not our job), we ask about the default gateway,  yes that set correctly.  ask about network mask.
we determine not correct. the 172.16.142 clients has a 255.255.224.0 netmask rather than the correct 255.255.255.224.  no wonder it did't work, so the client
rings up the hosting and demand to know why the netmask changed.  gotta run getting paged again
:professorcat:

My Moral Fibers have been cut.

deanwebb

#69
October 23, 2018, 11:53:36 AM
Ouch. Netmask screwed up.

Was this an automation problem or a bad admin problem?
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

icecream-guy

#70
October 23, 2018, 03:08:44 PM Last Edit: October 23, 2018, 03:13:46 PM by ristau5741
Quote from: deanwebb on October 23, 2018, 11:53:36 AM
Ouch. Netmask screwed up.

Was this an automation problem or a bad admin problem?

I don't know. I disconnected when the netmask issue was discovered, not a firewall issue.

They said it did work, but I don't think it ever did.

I don't know which is worse, apps guys  that lie, or apps guys that  know zilch about networking and how their apps work.
:professorcat:

My Moral Fibers have been cut.

Nerm

#71
October 31, 2018, 03:09:48 PM
When your all like...
:notthefirewall:

...but then it is the firewall.
:frustration:

deanwebb

#72
November 05, 2018, 04:12:29 PM
^ I feel your pain. It can not be the firewall 999 times out of 1000, but your credibility's blown on that one time that it is...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Otanx

#73
November 05, 2018, 04:39:12 PM
99 of the 100 times it is the firewall it actually is someone failed to submit the change ticket correctly. Firewall is configured per change ticket. Go get your "new" change approved, and we will make it work.

-Otanx
Print
Go Up Pages 1 ... 3 4 5
User actions