Sky Is Dark, Network Dark, Phone is Lit Up

Started by deanwebb, June 10, 2015, 04:18:01 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions

deanwebb

June 10, 2015, 04:18:01 AM
 :problem?:

Yay I have the on-call phone and I can confirm that the phone works just fine, even at 4AM...

:glitch:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

FilipiNomad

#1
June 10, 2015, 05:14:33 AM
LOL I feel your pain.  What is your rotation like?

SimonV

#2
June 10, 2015, 06:17:50 AM
I've been on Standby for almost a year now, one week every month. The first few months I had calls regularly but for the last six months I haven't received any.
Lots of stuff was off-shored though, so maybe they just don't know the number at the new helpdesk. But I'm not complaining  8)

Netwörkheäd

#3
June 10, 2015, 08:32:15 AM
Quote from: FilipiNomad on June 10, 2015, 05:14:33 AM
LOL I feel your pain.  What is your rotation like?
One week every 7 weeks. We used to have a follow the sun model where we would have weeknights off, but now we're back to a "just guys in the main US location" model.

Thankfully, WAN provider knew how to issue a shut / no shut power combo and the link is back up. Which is good because the backup link doesn't work right now.
Let's not argue. Let's network!

Reggle

#4
June 11, 2015, 04:19:43 PM
I had once every 8 weeks. Now I'm off the hook. Don't miss it...

deanwebb

#5
June 12, 2015, 10:43:38 AM
Another major WAN outage... looks like the cause of this one as well as Wednesday's was a *fiber cut due to construction*.

:rage:

The on-call phone is currently going through massive gyrations from all the Orion text updates flowing in from the Orion monitor that lost connectivity to all its nodes... and just now had that connectivity restored...

:kiwf:

The WAN is back up, though, so it should be a few hours yet before the backhoe operator finds the fiber and cuts it again. I will be able to get lunch.
:joy:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

#6
June 12, 2015, 02:00:54 PM Last Edit: June 12, 2015, 02:02:53 PM by SimonV
How critical is that site? Not fully diverse when it comes to WAN connections? For production or critical sites, we always insist on having physically redundant paths just for that reason. Those construction guys eh  :twisted:

This one guy that broke a fiber, he was gonna fix it with this



:awesome:

deanwebb

#7
June 12, 2015, 02:36:51 PM
It's only the main data center for the Americas. That's all. We only lose one-third of our entire operation when it goes down.

Thing is, this thread is also connected to the "WAN Provider Experiences" thread... we have WAN provider issues AND construction issues at the same time... yeesh.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#8
July 09, 2015, 11:35:25 PM
New fun... I'm already up to do a presentation for APAC, so I might as well stay up for when Europe comes online so I can talk to a team there to resolve an issue with our certs that's been killing our wireless 802.1X with mobile devices.

Sometimes "follow the sun" support hours happen to you, even though you're not following the sun yourself. I'm going to be soooooo messed up in about 12 hours, when I finish my last call for the day. :partay:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

#9
July 10, 2015, 02:03:38 AM
Yes, Europe just got out of bed. How can we help?

mmcgurty

#10
July 10, 2015, 07:22:12 AM
I do on-call every 4 weeks for 1 week and every 4 weeks for 1 week as a backup on-call.  My last rotation just about killed me due to working on other projects in addition to on-call problems.  I have been at the same place since 2008 and it was probably the worst on-call rotation I have done since I started in my early days and knew nothing of the environment.  I couldn't hand off the on-call phone quickly enough on the rotation day.  I was really glad to go to sleep and not get bothered.  We added a new guy about a month ago, I can't wait until we add him to the on-call and we go to a 5 week rotation.

deanwebb

#11
July 10, 2015, 08:26:20 AM
Quote from: SimonV on July 10, 2015, 02:03:38 AM
Yes, Europe just got out of bed. How can we help?

Can you destroy all iPhones? Then they'd all go back to using laptops, like proper IT professionals.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

that1guy15

#12
July 10, 2015, 09:04:49 AM
Currently once every 8 weeks for me. Its nice here as help-desk and desktop support field most issues so its usually pretty quiet. I feel for those desktop guys though.

My only problem with is though is the on-call schedule starts on Friday. Why Friday...
That1guy15
@that1guy_15
blog.movingonesandzeros.net

deanwebb

#13
July 10, 2015, 09:47:15 AM
Our oncall goes from Tuesday to Tuesday, which means I have to rearrange my life for oncall, since I work from home most Tuesdays.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Nerm

#14
July 10, 2015, 12:14:07 PM
My on-call schedule kinda goes to one of those "problems of a company of a certain size". Being a small company I am the on-call rotation lol. We do have one other guy that takes emergency calls so I guess I can say I am 1/2 the rotation. :partay:
Print
Go Up Pages1
User actions