WAR STORIES!

[deanwebb]

Recently had this fall in my "projects" folder.

Client bought new server running Windows Server Std 2012 R2 and is having trouble getting AD migrated from old server. *note: Onsite only as old server is not connected to the internet.

I get onsite and find that the old server they are trying to migrate from is running Windows Server 2000. WTF? This client has 3 full-time in-house IT personnel and not one of them thought to check the migration path. Worse why are they still running a 2000 server to begin with?

[scottsee]

I haven't been here much but there were a couple of things I've had to crawl through recently.

We have 2 x 300mb replication circuits for DRBCP backup & duplication. In July I started noticing they were heavy unbalanced and CIFS replication traffic was coming to a crawl for no reason. fast forward 6 weeks and endless testing I nailed it down to our dell open storage DR4100 appliances. The replication buffer was written to use best effort shared memory (long story). They wrote us a patch that created a round robin replication buffer for unequal link speed within 3 days. Just one of those things.

Replication speeds on a 20Gb teams were passing at a speed of 4-5Gb per flow across our VCF.. Turns out the 45drive storinator storage device we purchased 11 month ago came with a supermicro motherboard sporting 2 x PCIe 2.0 x4.. Apparently the Intel X540 dual 10GB cards require PCIe 3 x 8.. So were stuck at a 10Gb max flow rate across that one

Setup some new Dell M1000e bladecenters last week, stacked IO aggregator's dual quad 10Gb base-t cards, 80Gb LACP on our QFX switches.. Tested 31Gb/s across the fabrics today on the new blade servers..

It's 4pm on a friday!!!

[deanwebb]

"Can we export a private key and send it to a vendor so he can decrypt Wireshark captures?"

[icecream-guy]

"Can we export a private key and send it to a vendor so he can decrypt Wireshark captures?"

well ya can, if you want to trash the while PKI, generate a new private key and public key, then whoever asked this silly question has to contact everyone with the old public key and provide them with the new public key. 

[Otanx]

"Can we export a private key and send it to a vendor so he can decrypt Wireshark captures?"

Not sending to a Vendor, but I have made our server teams give me private keys to decrypt traffic when they try to blame the network. One of our security guys didn't know this was a thing, and his head exploded the first time he saw me do it. Also note our normal procedure is create new keys, import them, do troubleshooting, gen new keys, import those. Then I only get keys to decrypt traffic for a short time.

The few times I have gone this far it is after they refuse to believe me that the network is fine, and tell management they can't meet a deadline because the network team can't fix the network. Then I have to go into "Do the server teams job for them" mode. 

-Otanx

[deanwebb]

"Can we export a private key and send it to a vendor so he can decrypt Wireshark captures?"

Not sending to a Vendor, but I have made our server teams give me private keys to decrypt traffic when they try to blame the network. One of our security guys didn't know this was a thing, and his head exploded the first time he saw me do it. Also note our normal procedure is create new keys, import them, do troubleshooting, gen new keys, import those. Then I only get keys to decrypt traffic for a short time.

The few times I have gone this far it is after they refuse to believe me that the network is fine, and tell management they can't meet a deadline because the network team can't fix the network. Then I have to go into "Do the server teams job for them" mode. 

-Otanx

Already there with you about doing the server guys' jobs...

[icecream-guy]

not just the server guys, it's doing everyone else s job, just to prove it's not the network..

[Hunterman1043]

Fun little tidbit.

Team lead - "People are having issues with the SVN."
Me - "SVN is fine; server is up. Did they at least try to checkout the repo?"
Team lead - "Lemme ask... {5 mins later} Uhh, Nope."

[deanwebb]

"We have to reduce headcount, but want to keep the same project schedule. You think you can manage that?"

[Hunterman1043]

So this happened this past Friday.

I work at a local community college as a part-time tutor in the game department. I'm helping the current ProjectDev class create a game as their Technical Director; I basically am in charge of the server and the svn repo. I host it, I maintain it and I also help anyone in the class with issues they have with the game engine inside said repo. I have it set up to where people use a standard key pair to tunnel over ssh to the repo through a combination of Putty and TortoiseSVN clients in order to not have to deal with passwords. This server has been live for about 3 weeks or so. The reason we decided to go this route was because of 4 main reasons; control, 24/7 access, cheap, and for flexibility.

So now that that is out of the way, here is the actual story.

We start class on Friday morning and people are connecting to the server without issues. Everything is fine up until around 11 when a few students are reporting that they can't commit their changes. Other students start to report that they can't download the repo at all. Something is wrong. I check all the logs I can think of and find no issue server-side. A few minutes later, my boss and the tech guy for our department, come into the class and go directly to the instructor. They, basically, say that they just got an urgent email AND a phone call from the head of the DISTRICT-WIDE networking office accusing someone in our room of hacking on the network. (lol) They have a list of IP addresses for the computers that were reporting this issue. We inventory those computers and find that low and behold those were the same computers having issues committing. The instructor, the project lead and I set up a conference call with the guy and explain to him that we aren't doing anything nefarious and eventually we're able to get the computers unblocked. We were also able to whitelist my IP so that we won't have this issue in the future so long as I can get my server behind a static IP proxy of some sorts.

First of all, how did it take them 3 weeks to find my server? lol Second of all, why do they care about people SSHing out of the network? Finally, I'm pissed at the Instructor because he could've just called the guy on day 1 and explain the situation and this embarrassment never would've happened.

[deanwebb]

1. It took that long because they had to place the operation under observation, to make sure you weren't connected to radical moose or lambs.

2. They care because everyone in education cares. Don't you care? I used to care when I was teaching.

3. Agreed, the guy should have informed others about what was going on so you wouldn't have a skunkworks going on.

[Otanx]

3 weeks is quick. I don't remember exactly, but typical time frame for a company to detect a breach is over 200 days. They care because that SSH session could have anything in it. It bypasses any DLP or scanning they are doing to prevent data theft. Of course they should know their network, and I doubt you have credit card or PII data on random classroom computers, but that is the idea. IF they knew the network better they could have figured that out.

-Otanx

[mlan]

not just the server guys, it's doing everyone else s job, just to prove it's not the network..

QFT

[Hunterman1043]

typical time frame for a company to detect a breach is over 200 days.

That's comforting.

[deanwebb]

typical time frame for a company to detect a breach is over 200 days.

That's comforting.

Well, that's for breaches that are detected. Undetected breaches can go on being undetected for much longer than that.