Sky Is Dark, Network Dark, Phone is Lit Up

Started by deanwebb, June 10, 2015, 04:18:01 AM

Previous topic - Next topic

deanwebb

 :problem?:

Yay I have the on-call phone and I can confirm that the phone works just fine, even at 4AM...

:glitch:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

FilipiNomad

LOL I feel your pain.  What is your rotation like?

SimonV

I've been on Standby for almost a year now, one week every month. The first few months I had calls regularly but for the last six months I haven't received any.
Lots of stuff was off-shored though, so maybe they just don't know the number at the new helpdesk. But I'm not complaining  8)

Netwörkheäd

Quote from: FilipiNomad on June 10, 2015, 05:14:33 AM
LOL I feel your pain.  What is your rotation like?
One week every 7 weeks. We used to have a follow the sun model where we would have weeknights off, but now we're back to a "just guys in the main US location" model.

Thankfully, WAN provider knew how to issue a shut / no shut power combo and the link is back up. Which is good because the backup link doesn't work right now.
Let's not argue. Let's network!

Reggle

I had once every 8 weeks. Now I'm off the hook. Don't miss it...

deanwebb

Another major WAN outage... looks like the cause of this one as well as Wednesday's was a *fiber cut due to construction*.

:rage:

The on-call phone is currently going through massive gyrations from all the Orion text updates flowing in from the Orion monitor that lost connectivity to all its nodes... and just now had that connectivity restored...

:kiwf:

The WAN is back up, though, so it should be a few hours yet before the backhoe operator finds the fiber and cuts it again. I will be able to get lunch.
:joy:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

#6
How critical is that site? Not fully diverse when it comes to WAN connections? For production or critical sites, we always insist on having physically redundant paths just for that reason. Those construction guys eh  :twisted:

This one guy that broke a fiber, he was gonna fix it with this



:awesome:

deanwebb

It's only the main data center for the Americas. That's all. We only lose one-third of our entire operation when it goes down.

Thing is, this thread is also connected to the "WAN Provider Experiences" thread... we have WAN provider issues AND construction issues at the same time... yeesh.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

New fun... I'm already up to do a presentation for APAC, so I might as well stay up for when Europe comes online so I can talk to a team there to resolve an issue with our certs that's been killing our wireless 802.1X with mobile devices.

Sometimes "follow the sun" support hours happen to you, even though you're not following the sun yourself. I'm going to be soooooo messed up in about 12 hours, when I finish my last call for the day. :partay:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

Yes, Europe just got out of bed. How can we help?

mmcgurty

I do on-call every 4 weeks for 1 week and every 4 weeks for 1 week as a backup on-call.  My last rotation just about killed me due to working on other projects in addition to on-call problems.  I have been at the same place since 2008 and it was probably the worst on-call rotation I have done since I started in my early days and knew nothing of the environment.  I couldn't hand off the on-call phone quickly enough on the rotation day.  I was really glad to go to sleep and not get bothered.  We added a new guy about a month ago, I can't wait until we add him to the on-call and we go to a 5 week rotation.

deanwebb

Quote from: SimonV on July 10, 2015, 02:03:38 AM
Yes, Europe just got out of bed. How can we help?

Can you destroy all iPhones? Then they'd all go back to using laptops, like proper IT professionals.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

that1guy15

Currently once every 8 weeks for me. Its nice here as help-desk and desktop support field most issues so its usually pretty quiet. I feel for those desktop guys though.

My only problem with is though is the on-call schedule starts on Friday. Why Friday...
That1guy15
@that1guy_15
blog.movingonesandzeros.net

deanwebb

Our oncall goes from Tuesday to Tuesday, which means I have to rearrange my life for oncall, since I work from home most Tuesdays.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Nerm

My on-call schedule kinda goes to one of those "problems of a company of a certain size". Being a small company I am the on-call rotation lol. We do have one other guy that takes emergency calls so I guess I can say I am 1/2 the rotation. :partay: