Amazon EC2 Cloud is failing

[Ragns]

Apparently the so reliable cloud computing platform been down for a while. I'm so glad I moved our Minecraft servers from there.

Amazon’s EC2 cloud, used by thousands of major websites, crashed yesterday. The failure left users across the U.S., including Reddit and Foursquare, offline for up to 12 hours. The company said a glitch forced the system to create back-ups of everything in its storage volumes, jamming up storage capacity. Hacker group Anonymous, responsible for trying to crash the system late last year, said they are not responsible. All areas but one have been repaired.

In a similar, but unrelated incident, PlayStation users experienced outages yesterday, affecting 75 million gamers. The fix for this failure may take much longer than the Amazon snafu, with Sony representatives telling users to expect continued outages for "a full day or two." The server experienced similar malfunctions last month.

Both instances reveal the fragility of the online cloud, where reliability is king. For businesses that rely on elastic cloud computing services for revenue, Amazon’s hiccup may push customers to rethink their current external server models.

Source http://www.adweek.com/news/technolog...falling-130882

[DAKPluto]

Cydia is also included in this BTW, for you iphone jailbreakers.

[Chicomagiko]

and nothing of value was lost.

[Correction]

and nothing of value was lost.

This ain't spam, go away.

This pretty much cost my company ~18 hours of downtime, with several people working overnight to restore dbs from backups. Lost about 9 hours of data, but we're now up and running and at the very least looking to get some free cpu time from Amazon for what this cost. I dread to think what happened to less well-funded startups that can't eat the up-front costs of bringing up 6-8 volumes to replace the lost EBS accessibility.

[Correction]

More level-headed assessment of what happened:

http://justinsb.posterous.com/aws-do...sky-is-falling

Short term I see a lot of people dumping AWS altogether, even though it's probably not necessary. This was an EBS outage, and people fucking hated working with EBS. They'll probably stick to simple storage even though that's west-coast only and wait for a competitor with better availability zone policies (and enforcement thereof) before jumping ship.

11:54 AM PDT Quick update. We've tried a couple of ideas to remove the bottleneck in opening up the APIs, each time we've learned more but haven't yet solved the problem. We are making progress, but much more slowly than we'd hoped. Right now we're setting up more control plane components that should be capable of working through the backlog of attach/detach state changes for EBS volumes. These are coming online, and we've been seeing progress on the backlog, but it's still too early to tell how much this will accelerate the process for us.

For customers who are still waiting for restoration of the EBS control plane capability in the impacted AZ, or waiting for recovery of the remaining volumes, we understand that no information for hours at a time is difficult for you. We've been operating under the assumption that people prefer us to post only when we have new information. Think enough people have told us that they prefer to hear from us hourly (even if we don't have meaningful new information) that we're going to change our cadence and try to update hourly from here on out.

Some people are just plain desperate, or setting up a pass-the-buck feed of 'hey AWS hasn't fixed it not our fault' updates for their clients. I feel kinda sorry for them.

[Ragns]

Some people are just plain desperate, or setting up a pass-the-buck feed of 'hey AWS hasn't fixed it not our fault' updates for their clients. I feel kinda sorry for them.

Was funny to see the we're sorry if Amazon suck message on most of Zynga's *Ville games too.

[Chicomagiko]

Butthurt nerd much? It's quite the shame when companies don't have contingency back up plans or off grid solutions. Its been a while since I worked in "IT" but this is just LOL.

[Correction]

It's quite the shame when companies don't have contingency back up plans or off grid solutions. Its been a while since I worked in "IT" but this is just LOL.

Such 'back up plans' come with substantial startup and upkeep costs, something AWS is supposed to have your back on. Having a 'back up plan' for an AWS outage of this magnitude was possible for say, Netflix, who have enormous manpower and money to throw at their own distributed storage network failover. It's less practical for the smaller businesses that like the pricing and guarantees of EC2 and shouldn't be taken as a sign of carelessness. Come back when you know what the fuck.