Skip to main content ↑ ↓
Topic: Forum network down this morning (Read 1088 times) previous topic - next topic

Forum network down this morning

Now if that didn't cause a bunch of heartburn, we don't know what will!

Our hosting company had a network server problem overnight/this morning.  They are still assessing the situation, but we appear to finally be back up and running.

Pass the aspirin and the Pepcid A/C....

Michelle and Steve
Learn every day, but especially from the experiences of others. It's cheaper!  - John C. Bogle

2000 U320 36' non-slide / WildEBeest Rescue
2003 U320

Re: Forum network down this morning

Reply #1 –
I looked for it yesterday and did not see it. Said not connected to internet. I will say that it has been remarkably stable for a while. It is up much more than Yahoo was or is currently.
2025 Wanderbox Outpost 32 on F600 Expedition Motorhome
2015 Born Free Royal Splendor on Ford 550 nonslide version  for sale
Former Coaches  covering. 360,000 miles
1999 34 U270
2000 36 U320
2001 42' double slide U320
2018 Jeep Rubicon

Re: Forum network down this morning

Reply #2 –
As I'm sure some of you have noticed we continue to have problems with the site server going down.  It happened last night and again this afternoon. 

There has yet to be a determination as to what is causing the server to go down (by the provider/datacenter), but when it does it requires a hard power boot to come back to life.  All I can say at this time is its being worked on.
2000 / 36' / U320 / WTFE
WildEBeest / "Striving to put right what once went wrong"

Re: Forum network down this morning

Reply #3 –
Quote from: Steve –
As I'm sure some of you have noticed we continue to have problems with the site server going down. 

There has yet to be a determination as to what is causing the server to go down (by the provider/datacenter), but when it does it requires a hard power boot to come back to life.  All I can say at this time is its being worked on.

Just a quick update - we apologize for the continued server problems.  Steve is working with the provider to try and understand what's going on.  Last night, we had hosting moved from the "factory" server in the data center to the "cloud" hosting to try and get away from the hardware that was crashing.  Unfortunately, the move was rather bumpy and Steve is still fixing some parts of the forum.  It's also why many of you may have gotten errors last night.

To add insult to injury, the problem seemed to tag along with one of the other 5 sites that were moved "to the cloud" along with us.

I know it's a small comfort, but this is a highly-rated, not oversold hosting company and we're dealing with large data centers.  We are not the only ones affected.  Still, it's very frustrating to us and Steve is doing his best to address forum availability and stability.

Michelle
Learn every day, but especially from the experiences of others. It's cheaper!  - John C. Bogle

2000 U320 36' non-slide / WildEBeest Rescue
2003 U320

Re: Forum network Problems

Reply #4 –
I want to provide an update on the server / hosting issues we are facing.  I really don't have as much information as I'd like but here is what I know.

We were originally on a server, the same one that we have been on for months with very good performance, that began to experience problems with the raid 10 array. (disk storage for the non geeks). 

The "plan" in the next few months was to transition the sites from this server to a cloud server configuration.  This is a buzz word that basically means redundancy on servers and storage that should equate to better uptime and IF things are balanced correctly response that is better or equal to what we had.  Of course for the provider it means higher utilization of hardware so it lowers their cost but the hosts don't spin it that way ;)

So since the old server was a bit wobbly some sites were moved to the cloud ahead of time, we were one of those.  There were a couple of bumps but at first but things were generally good.  However the old servers problems continued so more sites were pressed in to the cloud system.  The cloud system was not ready for the load that all of these moved sites placed on it, and I believe its actually disk I/O bound right now.  The system admins have made some bad decisions in how to resolve this, the result of which we have all felt over the last 24-48 hrs at least.

So whats next?  There is going to be more sporadic outages that much I know.  Although this host has been very good to date and does have a very good reputation (its a small business and not a one of the 'super' host sites), I am reviewing our options to move to another provider as even when this gets fixed I would remained concerned about their ability to address another major hardware issue.
2000 / 36' / U320 / WTFE
WildEBeest / "Striving to put right what once went wrong"

Re: Forum network down this morning

Reply #5 –
And the madness continues !

After moving to the "cloud" ... aka storm cloud, there were performance issues that the provider (and customers) were not happy with.  Some of you may have noticed the system had noticeable performance swings, moving from very responsive at times to sluggish other times.  I suspect this is a result of the cloud data structure and the dynamic nature of how the forum pages are built (read I/O bottleneck)  and there was simply a tipping point where the performance did not degrade gracefully and instead just fell off.

So the solution? ... well if you are reading this you are on the solution!  Not sure if its back to the future or a Tardis trip with Dr. Who to the past but we are back to the server setup we had before we began this wild ride.

Once again I believe I have everything set up, and have now moved all the posts and data to the new/old server.  This move should have been generally transparent but you may have noticed a gremlin or two during the day.  If you find some things not working please let me know.
2000 / 36' / U320 / WTFE
WildEBeest / "Striving to put right what once went wrong"

Re: Forum network down this morning

Reply #6 –
** Notice **

For the nite owls ... tonight (2/5/11) @ 10:30PM PST, the server will be down for ~30mins so the processors can be upgraded.  After the upgrade the server will be a dual quad core harpertown (total 8 cores) with 12Gigs Ram, 4 15k  Sas drives running hardware Raid10.
2000 / 36' / U320 / WTFE
WildEBeest / "Striving to put right what once went wrong"