Current frustration...

Started by deanwebb, September 08, 2015, 10:09:38 AM

Previous topic - Next topic

deanwebb

Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

Proxy server down, which team gets the tickets?

:notthefirewall:

deanwebb

Heard about some Petya-related action at one of the big companies it hit... very very ugly situation there. My sympathy goes out to anyone hit by it, as it apparently wasn't just a WannaCry latecomer, but had some clever twists in it.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

wintermute000

When I was in large MSP, at least 2 days a week the entire security tower would be running around cleaning up after some crypto attack on a big account. That was a few years ago as well it's likely worse now, not to mention it was a Cisco wonderland and you all know how good ASAs are at sand box/IPS etc LOL

deanwebb

Quote from: wintermute000 on June 30, 2017, 05:31:24 PM
When I was in large MSP, at least 2 days a week the entire security tower would be running around cleaning up after some crypto attack on a big account. That was a few years ago as well it's likely worse now, not to mention it was a Cisco wonderland and you all know how good ASAs are at sand box/IPS etc LOL

Checking on the Cisco ASA after a crypto attack:

:shock:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#185
OK, one site in Latin America... Skype works just fine, but all web traffic does not. Outlook and Oracle don't work. AD works just fine. RADIUS, SNMP, and doing a telnet to port 80 or 443 on internal IP addresses work fine, but telnet to 80/443 for external addresses do not work. Proxy can be pinged and we can telnet to the proxy port, but we can't get the proxy script to run on these boxes... other sites just fine...

:zomgwtfbbq:

Update: HTTPS to WLCs works just fine, all around the globe.

:phone:

You heard right. HTTPS to anything but a WLC fails.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

Made progress: the crazy stuff may be linked to internal sites having external web certificates and the proxy not working properly for that site.

:notthefirewall:

Feelin' good it's not the firewall. Also not the NAC.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

Confirmed it was the proxy by switching the users to a legacy proxy that's about to be decommissioned. That proxy then was able to proxy the traffic to the proxy in production. Now I need to hand this off to the proxy team.

:yeahright:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

Turns out, it was NOT a proxy issue... now I have to start a thread about what fixed it so I can learn more stuff better.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Otanx

You can only have one priority one. No you can't say priority 1a and 1b. That is priority 1 and 2. No I can't work on both at the same time. They each take 4 people, and I have 6. Grow a backbone and tell one customer they have to wait. It isn't my fault both customers waited till the last minute to ask for help, but apparently it is my problem. Also every time you bug me about this puts me further behind because you wanted a 30 minute meeting about 1, and then another 30 minute meeting about 2, and you want to schedule these with a 30 minute opening between them in case the first one goes over. This means I am getting nothing done for an hour and a half.

Also if these two are so important stop interrupting my guys to do something else that "just takes 5 minutes". By the time they save what they were doing, login to do your "5 minute" task, then go back to the original task, remember where they were, and get back to that priority 1 it has been 15 - 30 minutes.

The best part was getting about 50% done with priority 1 (which had 100% to be done by COB Friday or heads would roll), and asking the customer for some information we need that wasn't in the request. The answer came back that they will get that info to us on Monday. We made it clear we need the info to finish the tasks. Yep, we just are going to push the deadline to end of next week.

I feel better. Yesterday kind of sucked. Now that one is on hold the other "priority 1" should be done on time. It will be close, but as long as the data center doesn't catch on fire we should be good. (hmmm, why is my temp sensor reading 300 degrees?)

-Otanx

deanwebb

"If you keep talking, it's gonna take longer..."
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

dlots

#191
Me working with a client who "Passed" the CCIE written on a packet cap.  He is obsessed with TCP "Zero Window" packets.  If you are not familiar with them that's when one device tells the other that it is busy and can't take any more data.

We started the capture, pinged the device, then ran the test.

Client:
look at 192.168.2.205
(You can see laptop trying to connect to something that wouldn't reply)
Me:
the pings are at packet number 11519, this is long before we started the test.
[....]
Client:
so here is my concern any hiccup between client and server will give the program an error. We need to look at how we can stop instances where the communication is lost in order to stop the timeout error.
Me:
I don't see where the communication is lost though, can you point me to something??
Client:
look at packet 199
Me:
that happened WAY before the test actually started, it's the wrong server, and it happened after the TCP session was closed


Notably the issue was also a Zero Window which isn't a network issue.

deanwebb

So, he's CCIE Written, But Expire?
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

wintermute000

I love it when people tell me they passed the written. I just straight up ask them if they've passed the lab and then when they're planning on attempting it

deanwebb

Quote from: wintermute000 on July 10, 2017, 04:48:47 PM
I love it when people tell me they passed the written. I just straight up ask them if they've passed the lab and then when they're planning on attempting it

We had a guy from India apply to us who had CCIE-Written. We asked and he said it took him a year and a half to prepare for it and then he went to the lab and failed it. He then said he didn't want to attempt the lab again and got into other vendors instead.

He had no hesitation as we asked him questions about said other vendors' stuff and he really knew the products well on a technical level. Guy got thumbs up from us.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.