OUTAGE REPORTS

Started by deanwebb, May 10, 2015, 08:42:12 AM

Previous topic - Next topic
Print
Go Down Pages1 2 3 4
User actions

deanwebb

May 10, 2015, 08:42:12 AM
It's not that these should never happen, but that they should never happen without an explanation.

10 May 2015 - less than a day (I think) due to a database table needing repair. Affected Tapatalk users so they could not log in. PC users could log in, but with an error at the bottom of the page. SimonV reported it to me via LinkedIn and I got on it right away. Repair executed, all log ins seem normal with no errors.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#1
May 26, 2015, 02:44:05 PM
26 May 2015 - Had one today for a few minutes, starting between 2:00 PM US Central time (when I checked before getting a snack) and 2:30 PM US Central time (when I got back from getting my snack and noticed the site was down). Outage ended by 2:40 PM US Central time. Given that my other websites with this host were down, it was either a connection to the host or a host's server that lost its mind temporarily. Hardly worth mentioning, except for the fact that it was an outage, so it counts.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#2
February 26, 2016, 11:11:59 AM
No outage, just wanted to let everyone know I upgraded to the current SMF version today.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#3
March 07, 2016, 11:44:32 AM Last Edit: March 07, 2016, 11:49:24 AM by deanwebb
Nf_sessions table took a dump today around 8:42 AM my time, roughly 3 hours ago. It looks to be all right now after I ran a repair operation on it.

:whatudo:

Just pushed a few buttons marked "repair" and "check" until all the lights turned green.

For some reason, I now want layer 2 adjacency for major datacenters around the world. I just do.

:mssql:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

SimonV

#4
March 07, 2016, 01:23:40 PM
Quote from: deanwebb on March 07, 2016, 11:44:32 AM
Just pushed a few buttons marked "repair" and "check" until all the lights turned green.

Good work, have you ever considered becoming an MS SQL developer? :)

icecream-guy

#5
March 07, 2016, 01:25:25 PM
at least googling the error passed the time....
:professorcat:

My Moral Fibers have been cut.

deanwebb

#6
March 07, 2016, 01:25:55 PM
Quote from: SimonV on March 07, 2016, 01:23:40 PM
Quote from: deanwebb on March 07, 2016, 11:44:32 AM
Just pushed a few buttons marked "repair" and "check" until all the lights turned green.

Good work, have you ever considered becoming an MS SQL developer? :)
Yeah, but I'd rather be paid for turning things off and on again and running ping commands.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#7
June 29, 2016, 08:07:51 AM
10PM June 28 - 8AM June 29

Issue at host: rebooting devices worked.

:itcrowd:
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

icecream-guy

#8
June 29, 2016, 11:00:02 AM Last Edit: June 29, 2016, 11:02:06 AM by ristau5741
Quote from: deanwebb on June 29, 2016, 08:07:51 AM
10PM June 28 - 8AM June 29

Issue at host: rebooting devices worked.



Didn't the monitoring team escalate the issue to the on call guy ???
:professorcat:

My Moral Fibers have been cut.

deanwebb

#9
June 29, 2016, 11:32:46 AM
I'm both monitoring and on-call. I monitored this morning and then called my host. :)

Sent from my SM-N900P using Tapatalk

Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#10
July 20, 2016, 06:28:25 PM
Big outage... just got back up after a week offline. The host, a small operation I've used since 1998, took a lightning strike in a major storm and his line went totally out. Telco in his area delayed service and finally got to him today. I am very, very sorry for the disruption and, believe me, I have missed dearly being able to hit these forums daily.

My host was going to be leaving the hosting business, anyway, and he was planning to let his clients know a day or two after the outage hit. I'm very thankful to all the work he's done behind the scenes for this and my other websites, and I wish him well as we wrap up our business relationship.

Going forward, I am looking to move the forums to a new host on the lines of Rackspace, Dreamhost, and the like. Good news is that if we move to Linux hosts, I can turn on SEO for the site.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

wintermute000

#11
July 20, 2016, 08:04:14 PM
Good work and thanks

deanwebb

#12
July 20, 2016, 08:08:45 PM
Set up an inquiry with Dreamhost. They offer a free SSL cert from Let's Encrypt! which is a nice thing. Be nice to get all SSL-y in here.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#13
July 22, 2016, 02:06:50 PM
Planned outage this weekend... going to move to the new hosting environment. I'll start on Saturday afternoon around 1PM (Dallas, TX time) and should be finished within 12 hours or less. I'll send out an email to everyone once the site is up and running in its new hosting location.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

deanwebb

#14
July 22, 2016, 08:59:18 PM
Moved my personal sites tonight, and zzzptm.com is working just fine. aohell.com needed some more tinkering, but I think I'll be able to have it at 100% later on. Just have to let stuff propagate.

I'll put it into maintenance mode tomorrow, download the database, change the DNS records, restore the database to the new host, make sure the .htaccess file points to index.php and then let the DNS propagate. I've already got the forum files loaded on the new host and the mysql stuff is very easy to set up.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.
Print
Go Up Pages1 2 3 4
User actions