catastrophic failure [updated: Recovery!]

[UPDATE: Nov. 12, 7:30AM Pacific]

We were able to recover the month-long lost data and are now working to restore it on a new server. We will be down as far as new user contributions for a few hours, but will be back with full functionality later today. JJ is working on transferring all the new submission that happened in the past day or so. However, as we warned before, most comments (made in the last day) will be lost.

Backround

So, for a while now (pretty much since the elections) we've been trying to move to a different hosting setup. We use Rackspace which has a very high reputation for service and support in the industry. Rackspace has a new cloud service called Mosso. We moved there earlier this summer, but due to a large variety of technical problems, it didn't work out for us. So we moved back to regular Rackspace hosted server.

When we moved back, about a month ago, our new Rackspace service told us they can't support backup operations until we switch to a new operating system (Redhat from CentOS) which we were scheduled to do about 2 weeks from now. But until then, we needed to do backups.

Earlier this week, another member of our team asked RS to add an external hard drive so we can have backups onto there until our new server is ready. When they went to add this hard drive, which was supposed to happen without any service interruptions, they reported that the existing drives had failed and recovery attempts were not successful. It's such a rare thing (for a Raid 5 system to have two hardware failures in minutes) that one person involved has already theorized that it might have been done on purpose. While we can't draw any conclusions at the moment, we are asking questions and pursuing the matter.

While Rackspace is working on trying to get the data off the old drives, which could take days. We are stuck with the last successful backup we did have, which was from October 10.

I hope I didn't give you a headache. JJ is ready to stay up all night trying to update the page with whatever he can. Please don't email him! It won't help. He will probably write in this blog and you can respond to him (soon.)

Our last updated page is cached at google.

Thank You for your continued patience and understanding.