Power failure caused outage
A full datacenter power failure just caused about 55 minutes of downtime.
Of course we weren’t able to alert you through the blog — it was down — so we’re doing to use a 3rd-party system for that starting immediately.
We’re also looking into what it would take to do some sort of (probably read-only) off-site backup that would allow us to at least continue basic service in the event of a complete physical disaster, but that will probably take a few months to come to fruition.
I won’t say “thanks for your patience and understanding.” Instead I’ll say: We’re going to start building out new ways of weathering even this sort of storm.
Thanks for informing us, I guess these things happen, can’t help but wonder why the datacenter’s backup power didn’t kick-in? I think you use Global IP Networks don’t you? they make a huge deal on their Web site about UPS and there Generators and how power failures won’t ever happen, what went wrong?
We don’t know yet what went wrong. I think they’re still just trying to get everything working again before they declare.
We use FireHost.
Why not colo at a facility where you trust their power situation or use Amazon, etc with multiple datacenters/ easy redundancy? FireHost seems like a bit of redundancy considering you’re the “host” of our blogs who is hosting on another hosting provider who is probably colocating? A lot of layers to keep track of. We’ve been Colo’ing our servers with ForTrust in Denver for 4 years for our main ecomm website and they have never had a single power outage in their company history. This was one of the main reasons we chose them. eBay and a few others are there too for the same reasons. Highly recommended if you don’t want Amazon-ish cloud hosing, which is great too and even simpler.
Don’t they have backup generators?