Current frustration...

Started by deanwebb, September 08, 2015, 10:09:38 AM

Previous topic - Next topic

deanwebb

Quote from: Otanx on March 22, 2019, 09:08:25 AM
Quote from: deanwebb on March 22, 2019, 07:11:40 AM
I've got a customer that has about 30K endpoints, all on a flat 10.0.0.0/8 network. Because that's how they have their Meraki set up.

That means there are two of these out there. We have a customer that does this, and they don't have Meraki so it can't be the same one. Just to add to the insanity they expanded their network to remote sites. Those remote sites overlap IP space of course. So they do a weird double nat thing with proxy-arp and "reserve" the remote site space in their /8 then the firewall NATs it to itself somehow to make routing work.

It is ugly and I am glad I only am responsible for the external POP there.

-Otanx


Wait, which address is real and which one is NAT? Someone gave me an IP to check out, but I have no idea where to start!

:morty:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Otanx

Frustration solved post. Had to look at this thread to see if I had ranted about this already, but didn't find anything. About 6 months ago there were a ton of changes here. We had one network guy leave for a better job, I got promoted to a new slot that was created on the architect team, and the company did a reorganization that split the network team into two separate groups. The guys picked for the other team moved to another office, and are no longer part of the on-call roster. Basically we went from a team of 6 to a network team of 3, and one of those 3 is my old slot that isn't filled yet.

We just found out today that they added another slot to the team. So we now have two openings. Once we can fill them both we might actually be able to do more than keep the network from crashing, and won't have anyone rage quiting over the frequency of on-call.

Now to find someone that doesn't lie on the resume.

-Otanx

Otanx

Trying to do temperature monitoring. Dig into SNMP and find Cisco exposes several temperature sensors, and it is easy to get the temps. Also exposed is the "shutdown" temp for each sensor that if hit the switch will shutdown to protect itself. What is not exposed? A warning value. If you login to the switch, and do a show environment temp there is a warning value shown. They just don't expose it in SNMP. All the other vendors I poked at have current, warning, critical exposed for each sensor. Did a walk on the entire tree on Cisco looking for the temperature shown in the CLI. So now I have to play guessing games. Is 70% of critical a good value? 80%? Who knows.

-Otanx

deanwebb

Wow, no warning value...  :naughty: Bad Cisco! Sounds like you need to submit a feature request.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Otanx

I am considering it. I know I will just get told about this great thing called DNA Center, or whatever the new cool Cisco monitoring tools are called. Last time I worked on it was called CiscoWorks.

Side note. We found a room with a failing AC. Inlet temps on a pair of switches were showing 40C. Critical to Cisco is 57C so we have a little time. Also you get a small scare when you finish with Cisco and start working on Arista, and see a temp of 347C. Then realize it is measured in 10ths of C, and is really 34.7.

-Otanx

deanwebb

347 C, I think, is the melting point of some soft steels... :)
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

config t

We had a contract turnover and lost our network engineer last week because the incoming company severely under-bid. They offered him a 30% pay cut, no PTO, no HOLA/COLA and a 7-day work week.

While I don't blame him for turning down the opportunity of a lifetime, as the only other network guy here my phone now rings a lot more for an extra $0/hour.

The replacement was promoted from within and is in hiding down south at the TCF. This individual doesn't even have a CCNA and "forgot" to come and do a turnover. Coooolio...
:matrix:

Please don't mistake my experience for intelligence.

Nerm

I'm sorry but I am having trouble understanding the 30% pay cut. I mean who the hell would think the guy would even consider it.

config t

Nobody who has an actual skill set would take a 30% pay cut. But I think that was the point.

I found out this morning that the outgoing engineer's tier 3 billet is now a "Tier 1 Network Engineer Support" billet. What exactly Network Engineer Support is, I have not a clue.

I think it means the individual can continue hiding down south at the TCF on night shift and run cable or something.
:matrix:

Please don't mistake my experience for intelligence.

deanwebb

Quote from: config t on October 21, 2019, 08:18:36 AM
They offered him a 30% pay cut, no PTO, no HOLA/COLA and a 7-day work week.


Did not know slavery was legal in Bahrain. :smug:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

Some thoughts I had: Two ways to perform changes during business hours and without change control:

1. To upgrade the code of a network device but don't initiate a reboot. Wait for power cut or crash or some other issue to initiate a reboot. Device loads new code next time  :mrgreen:

2. Apply config change but don't commit to "copy run start" until a later date. That way, if anyone complains you can say "that damned device, I'll reboot it". Proceed to reboot your device which will fix the issue and management will pat you on the back  :mrgreen:

config t

Respek. I too like to pull the occasional sneaky  :whistling: >:D
:matrix:

Please don't mistake my experience for intelligence.

deanwebb

Sometimes, one does have to have an aggressive interpretation of "standard change" in order to get things done...
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

icecream-guy

how about people that don't plan their projects,   spent 4 hours today to troubleshoot firewall issue only to find out the that the network wasn't trunked to the F5,  no wonder there were connectivity issues....   :blank:

and it was break fix, and they couldn't roll back,  something about some MS servers going EOL EOY :squint:
:professorcat:

My Moral Fibers have been cut.

icecream-guy

Quote from: ristau5741 on November 23, 2019, 03:12:58 PM
how about people that don't plan their projects,   spent 4 hours today to troubleshoot firewall issue only to find out the that the network wasn't trunked to the F5,  no wonder there were connectivity issues....   :blank:

and it was break fix, and they couldn't roll back,  something about some MS servers going EOL EOY :squint:

oh that was a good frustration.....
:professorcat:

My Moral Fibers have been cut.