Now if that didn't cause a bunch of heartburn, we don't know what will!
Our hosting company had a network server problem overnight/this morning. They are still assessing the situation, but we appear to finally be back up and running.
Pass the aspirin and the Pepcid A/C....
Michelle and Steve
I looked for it yesterday and did not see it. Said not connected to internet. I will say that it has been remarkably stable for a while. It is up much more than Yahoo was or is currently.
As I'm sure some of you have noticed we continue to have problems with the site server going down. It happened last night and again this afternoon.
There has yet to be a determination as to what is causing the server to go down (by the provider/datacenter), but when it does it requires a hard power boot to come back to life. All I can say at this time is its being worked on.
Just a quick update - we apologize for the continued server problems. Steve is working with the provider to try and understand what's going on. Last night, we had hosting moved from the "factory" server in the data center to the "cloud" hosting to try and get away from the hardware that was crashing. Unfortunately, the move was rather bumpy and Steve is still fixing some parts of the forum. It's also why many of you may have gotten errors last night.
To add insult to injury, the problem seemed to tag along with one of the other 5 sites that were moved "to the cloud" along with us.
I know it's a small comfort, but this is a highly-rated, not oversold hosting company and we're dealing with large data centers. We are not the only ones affected. Still, it's very frustrating to us and Steve is doing his best to address forum availability and stability.
Michelle
I want to provide an update on the server / hosting issues we are facing. I really don't have as much information as I'd like but here is what I know.
We were originally on a server, the same one that we have been on for months with very good performance, that began to experience problems with the raid 10 array. (disk storage for the non geeks).
The "plan" in the next few months was to transition the sites from this server to a cloud server configuration. This is a buzz word that basically means redundancy on servers and storage that should equate to better uptime and IF things are balanced correctly response that is better or equal to what we had. Of course for the provider it means higher utilization of hardware so it lowers their cost but the hosts don't spin it that way ;)
So since the old server was a bit wobbly some sites were moved to the cloud ahead of time, we were one of those. There were a couple of bumps but at first but things were generally good. However the old servers problems continued so more sites were pressed in to the cloud system. The cloud system was not ready for the load that all of these moved sites placed on it, and I believe its actually disk I/O bound right now. The system admins have made some bad decisions in how to resolve this, the result of which we have all felt over the last 24-48 hrs at least.
So whats next? There is going to be more sporadic outages that much I know. Although this host has been very good to date and does have a very good reputation (its a small business and not a one of the 'super' host sites), I am reviewing our options to move to another provider as even when this gets fixed I would remained concerned about their ability to address another major hardware issue.
And the madness continues !
After moving to the "cloud" ... aka storm cloud, there were performance issues that the provider (and customers) were not happy with. Some of you may have noticed the system had noticeable performance swings, moving from very responsive at times to sluggish other times. I suspect this is a result of the cloud data structure and the dynamic nature of how the forum pages are built (read I/O bottleneck) and there was simply a tipping point where the performance did not degrade gracefully and instead just fell off.
So the solution? ... well if you are reading this you are on the solution! Not sure if its back to the future or a Tardis trip with Dr. Who to the past but we are back to the server setup we had before we began this wild ride.
Once again I believe I have everything set up, and have now moved all the posts and data to the new/old server. This move should have been generally transparent but you may have noticed a gremlin or two during the day. If you find some things not working please let me know.
** Notice **
For the nite owls ... tonight (2/5/11) @ 10:30PM PST, the server will be down for ~30mins so the processors can be upgraded. After the upgrade the server will be a dual quad core harpertown (total 8 cores) with 12Gigs Ram, 4 15k Sas drives running hardware Raid10.