It's not that these should never happen, but that they should never happen without an explanation.
10 May 2015 - less than a day (I think) due to a database table needing repair. Affected Tapatalk users so they could not log in. PC users could log in, but with an error at the bottom of the page. SimonV reported it to me via LinkedIn and I got on it right away. Repair executed, all log ins seem normal with no errors.
26 May 2015 - Had one today for a few minutes, starting between 2:00 PM US Central time (when I checked before getting a snack) and 2:30 PM US Central time (when I got back from getting my snack and noticed the site was down). Outage ended by 2:40 PM US Central time. Given that my other websites with this host were down, it was either a connection to the host or a host's server that lost its mind temporarily. Hardly worth mentioning, except for the fact that it was an outage, so it counts.
No outage, just wanted to let everyone know I upgraded to the current SMF version today.
Nf_sessions table took a dump today around 8:42 AM my time, roughly 3 hours ago. It looks to be all right now after I ran a repair operation on it.
:whatudo:
Just pushed a few buttons marked "repair" and "check" until all the lights turned green.
For some reason, I now want layer 2 adjacency for major datacenters around the world. I just do.
:mssql:
Quote from: deanwebb on March 07, 2016, 11:44:32 AM
Just pushed a few buttons marked "repair" and "check" until all the lights turned green.
Good work, have you ever considered becoming an MS SQL developer? :)
at least googling the error passed the time....
Quote from: SimonV on March 07, 2016, 01:23:40 PM
Quote from: deanwebb on March 07, 2016, 11:44:32 AM
Just pushed a few buttons marked "repair" and "check" until all the lights turned green.
Good work, have you ever considered becoming an MS SQL developer? :)
Yeah, but I'd rather be paid for turning things off and on again and running ping commands.
10PM June 28 - 8AM June 29
Issue at host: rebooting devices worked.
:itcrowd:
Quote from: deanwebb on June 29, 2016, 08:07:51 AM
10PM June 28 - 8AM June 29
Issue at host: rebooting devices worked.
Didn't the monitoring team escalate the issue to the on call guy ???
I'm both monitoring and on-call. I monitored this morning and then called my host. :)
Sent from my SM-N900P using Tapatalk
Big outage... just got back up after a week offline. The host, a small operation I've used since 1998, took a lightning strike in a major storm and his line went totally out. Telco in his area delayed service and finally got to him today. I am very, very sorry for the disruption and, believe me, I have missed dearly being able to hit these forums daily.
My host was going to be leaving the hosting business, anyway, and he was planning to let his clients know a day or two after the outage hit. I'm very thankful to all the work he's done behind the scenes for this and my other websites, and I wish him well as we wrap up our business relationship.
Going forward, I am looking to move the forums to a new host on the lines of Rackspace, Dreamhost, and the like. Good news is that if we move to Linux hosts, I can turn on SEO for the site.
Good work and thanks
Set up an inquiry with Dreamhost. They offer a free SSL cert from Let's Encrypt! which is a nice thing. Be nice to get all SSL-y in here.
Planned outage this weekend... going to move to the new hosting environment. I'll start on Saturday afternoon around 1PM (Dallas, TX time) and should be finished within 12 hours or less. I'll send out an email to everyone once the site is up and running in its new hosting location.
Moved my personal sites tonight, and zzzptm.com is working just fine. aohell.com needed some more tinkering, but I think I'll be able to have it at 100% later on. Just have to let stuff propagate.
I'll put it into maintenance mode tomorrow, download the database, change the DNS records, restore the database to the new host, make sure the .htaccess file points to index.php and then let the DNS propagate. I've already got the forum files loaded on the new host and the mysql stuff is very easy to set up.
Since you're going to linux, is the forum software and hence layout changing? e.g. phpmybb like everyone else?
Software and layout change? NO
as in:
:no:
I'm copying the files straight over and it's SMF all the way, baybee!
This is me looking at the results of my first two sites' moves:
:greatoffer:
And then after I move this forum over and get it up, I will be OVER NINE THOUSAND
:goku:
OK, it is back up, looking good. We are ready for business again!
And if you're looking at my avatar and going,
:whatudo:...
I went with a new avatar for the new location, by the way. If you see Nezhualcoyotl looking to the right instead of Amitabh Bachchan looking to the left, you are at the correct location. :)
Looking things over, members likely will need to update their avatar info if it's not showing correctly. That's life in the big city, I'm afraid.
All the smilies loaded and are working, which is good.
:kidwoohoo:
Looks good from my other account, as well. Well done!
Had to do the right CHMOD on the attachments directory and fix the path on it to get our uploaded avatars back. Now they are back... Will go through, fixing other paths, but the boards are workable and postable to, so, yay.
:joy:
Well, it looks like I got most of the flags working. Also confirmed that it works in Tapatalk.
30 minutes today, Dreamhost gave me error 500 on this and my other sites. But, they are back up, so outage over.
Maintenance tonight by webhost, for 45 minutes starting at 10:30 US Pacific Time. We may be offline for some or all of that time.
This is routine Dreamhost maintenance, not related to anything in particular other than what IT has to do from time to time to keep the lights on.
Quote from: deanwebb on February 03, 2017, 10:31:14 AM
Maintenance tonight by webhost, for 45 minutes starting at 10:30 US Pacific Time. We may be offline for some or all of that time.
This is routine Dreamhost maintenance, not related to anything in particular other than what IT has to do from time to time to keep the lights on.
Please have your backups completed by 10:00 US PT. :wub:
Tuesday at 6PM PST, Dreamhost will migrate the database from the current server to a new one. Shouldn't be too bad.
:notbad:
Took zero outage in upgrading to 2.0.14, but I did make sure to backup the database, just in case.
Site was down for me yesterday morning, about the same time as this post I'm writing here now, except yesterday. I was busy so couldn't check back yesterday at all.
Working now though :)
Quote from: Dieselboy on July 12, 2017, 11:28:23 PM
Site was down for me yesterday morning, about the same time as this post I'm writing here now, except yesterday. I was busy so couldn't check back yesterday at all.
Working now though :)
That time is close to midnight, USA time, so maybe Dreamhost was doing maintenance... just a guess on my part. But I'm taking note of what you found and will keep an eye on things.
Had an outage for a short while today, due to a DDoS attack on DreamHost. All looks clear now.
Just upgraded to 2.0.15. Outage lasted for 0.0000000000000000000001 seconds, bet you didn't even notice it.
Quote from: deanwebb on February 01, 2018, 10:51:08 AM
Just upgraded to 2.0.15. Outage lasted for 0.0000000000000000000001 seconds, bet you didn't even notice it.
you are not meeting the 0.99999999999999999999999 up time requirement.
Quote from: ristau5741 on February 02, 2018, 06:38:28 AM
Quote from: deanwebb on February 01, 2018, 10:51:08 AM
Just upgraded to 2.0.15. Outage lasted for 0.0000000000000000000001 seconds, bet you didn't even notice it.
you are not meeting the 0.99999999999999999999999 up time requirement.
:rage:
Upgrading hosting to a virtual private server, since I was starting to max resources available on current hosting plan. Moves will take place behind the scenes during the day today, and should not be noticeable.
Basically, 118 times in the last month, we hit a max resources condition and could either not serve up a page in time or had to kill a process to get a page up. Going with the VPS option will get me dedicated resources and that will get us closer to our "23 nines" uptime goal.
We've had a lot of Internal Error 500 errors today. I've done some forum maintenance tasks and have opened up a ticket with Dreamhost. I'll keep you posted on developments.
I went ahead and also mentioned that I've had very slow loading times since going to VPS, so maybe they can fix that, as well.
Hmm... the 500s aren't happening now, and I'm wondering if the site response is now faster...
If so, well, I guess I should do maintenance tasks more often than once every 3 years... :whistle:
And now the site is like lightning fast! This is great! The tech support said,
"Checking your account I found the VPS and MySQL server were split between data centers. This split can cause latency issues. I have consolidated your servers into the same data center. You should now see a performance improvement for your sites."
Indeed, I do! Awesome!
Quote from: deanwebb on July 25, 2018, 07:28:02 PM
And now the site is like lightning fast! This is great! The tech support said,
"Checking your account I found the VPS and MySQL server were split between data centers. This split can cause latency issues. I have consolidated your servers into the same data center. You should now see a performance improvement for your sites."
Indeed, I do! Awesome!
gave up redundancy for speed, eh?
site does seem faster now.
They have backups, we don't need HA!
:steamtroll:
Upgraded to 2.0.17... not a lot of downtime, but I did want to acknowledge the change happened.
Small company change control FTW!
:problem?:
Outage yesterday: tech support from Dreamhost said to me:
Thank you for contacting DreamHost support. I'm very sorry to hear about
the issues with your site. It looks as if your VPS was hung (not
responding) so I went ahead and restarted it and this fixed the hung
state. Currently your sites are now loading but if you run into issues
like this in the future you may want to try restarting your VPS from the
panel (just make one request as multiple requests can cause issues). If
that doesn't fix the issue let us know and we can take a look for you.
So, next time, I'll know what to do. :D
Quote from: deanwebb on April 29, 2020, 09:40:01 AM
Outage yesterday: tech support from Dreamhost said to me:
Thank you for contacting DreamHost support. I'm very sorry to hear about
the issues with your site. It looks as if your VPS was hung (not
responding) so I went ahead and restarted it and this fixed the hung
state. Currently your sites are now loading but if you run into issues
like this in the future you may want to try restarting your VPS from the
panel (just make one request as multiple requests can cause issues). If
that doesn't fix the issue let us know and we can take a look for you.
So, next time, I'll know what to do. :D
sounds like a SOP to me.
Quote from: ristau5741 on April 29, 2020, 02:56:08 PM
Quote from: deanwebb on April 29, 2020, 09:40:01 AM
Outage yesterday: tech support from Dreamhost said to me:
Thank you for contacting DreamHost support. I'm very sorry to hear about
the issues with your site. It looks as if your VPS was hung (not
responding) so I went ahead and restarted it and this fixed the hung
state. Currently your sites are now loading but if you run into issues
like this in the future you may want to try restarting your VPS from the
panel (just make one request as multiple requests can cause issues). If
that doesn't fix the issue let us know and we can take a look for you.
So, next time, I'll know what to do. :D
sounds like a SOP to me.
Now that I know.
:oracle:
On 16 November, there will be an upgrade of the SQL backend during the Pacific Time night - 1AM to 7AM US Pacific time. So, that may mean a bit of inconvenience for our APAC members. Sorry about that. Should only be for a few minutes and then everything gets restored OK.
Planned outage:
On Thursday, August 11 we will be performing proactive maintenance on your MySQL server from 10:00 AM to 6:00 PM PDT. During this maintenance window we anticipate approximately 30 minutes of intermittent connectivity disruptions to that MySQL service. All other DreamHost services will remain unaffected.
There may be issues with the forums on the 11th, off and on. Please bear with. :)
Planned Outage:
I need to upgrade the forums to SMF 2.1.4 because there's a line of deprecated code that is filling up the error logs every time it gets hit. I think the upgrade will fix it. I'll be doing it probably Thursday night, because those logs fill up FAST and I don't want to get hit with another $15 per month to carry them. :)
So yeah, around 5-7PM Dallas time (US Central) on Thursday, gonna do some upgrading. Forums will be offline during that time until I say they're up again and if they act goofy for you during that time, that's why.
was the change submitted to that Change Management Team? Reviewed by the engineering and architecture boards? Have you run the upgrade scenario in the lab?
To be honest, I was kinda hoping there would be an unrelated outage somewhere that I could hang this off of and call this one an emergency change, then do the upgrade "for troubleshooting reasons" and use a generic admin account for direct access to do the upgrade so I'd have plausible deniability if anyone ever did a change request review.
Outage over, it will look differently for a little while until after I get a chance to do the packages.
Database conversion took WAY longer than planned, but the 3GB of error logs should be gone now.
Hosting outage happened today, hosts had a datacenter issue. It looks to be resolved now.
Had a wee tiny outage yesterday when the host introduced an error in the Apache configuration. It's now corrected and back up.
SSL Cert expired on 23 June and it should have been 23 August. Contacted support regarding the issue, should get a new cert soon.
SSL cert issue resolved.
By jing, that was lickety-split quick! :smug: