AWS S3 Outage

Saw an article in which Amazon claimed it was human error. Apparently some dude or dudette as the case man shut down the wrong server for maintenance. It caused a cascade effect. Wonder if they still have s job?

Kinet1c

paul78 wrote: »

Anyone having a bad day? Hope everyone's BCP plan is working as designed.

Yup, we'd a bad day with this but not catastrophic. We've a few items on our to do list after our internal post mortem. It was a late night considering we're in the EU.

Verities

DatabaseHead wrote: »

We are on Azure no complaints.

Historically, if you compare the amount of outages that Azure has experienced, you will see they occur more frequently with AWS.

DatabaseHead

Verities wrote: »

Historically, if you compare the amount of outages that Azure has experienced, you will see they occur more frequently with AWS.

We've been on Azure for over a year never experienced anything like this........

Verities

DatabaseHead wrote: »

We've been on Azure for over a year never experienced anything like this........

Let's take a quick look at the history of Azure's outages:

Global DNS outage hits Microsoft Azure customers | ZDNet

Microsoft Azure hit with widening outages in Europe and India - GeekWire

Azure Nightmare: Customer Suffers 9-Day Intermittent Outage, Gets No Help From Microsoft - Page: 1 | CRN

https://www.bloomberg.com/news/videos/b/689903ff-6758-411a-b834-44c813fd81c9

https://www.forbes.com/sites/benkepes/2014/11/18/its-a-return-to-the-azure-alypse-microsoft-azure-suffers-widespread-outage/#595c99d7421d

Microsoft Azure had more downtime in 2014 than main cloud rivals

UnixGuy

Let's put everything on the Internet...what could possibly go wrong?

DatabaseHead

Verities wrote: »

Let's take a quick look at the history of Azure's outages:

Global DNS outage hits Microsoft Azure customers | ZDNet

Microsoft Azure hit with widening outages in Europe and India - GeekWire

Azure Nightmare: Customer Suffers 9-Day Intermittent Outage, Gets No Help From Microsoft - Page: 1 | CRN

https://www.bloomberg.com/news/videos/b/689903ff-6758-411a-b834-44c813fd81c9

https://www.forbes.com/sites/benkepes/2014/11/18/its-a-return-to-the-azure-alypse-microsoft-azure-suffers-widespread-outage/#595c99d7421d

Microsoft Azure had more downtime in 2014 than main cloud rivals

Like I said WE never experienced an outage....... Completely satisfied with their service and most certainly never experienced an outage of this magnitude.

Amazon took a massive hit, they look like a joke at the moment and unreliable.

On a side note, the engineer, errr ex-engineer who screwed it up was released.....

http://gizmodo.com/amazon-says-one-engineers-simple-mistake-brought-the-in-1792907038

paul78

The Amazon post-mortem if anyone is interested - https://aws.amazon.com/message/41926/

alias454

DatabaseHead wrote: »

On a side note, the engineer, errr ex-engineer who screwed it up was released.....

Amazon Says One Engineer's Simple Mistake Brought the Internet Down

I read that article and didn't see where it mentioned the engineer in question was released?

paul78

alias454 wrote: »

I read that article and didn't see where it mentioned the engineer in question was released?

Yeah - I don't see a mention of it anywhere. It would be very unfortunate and a bit unusual if Amazon released the engineer for this type of mistake. And any disciplinary measures certainly wouldn't be something that Amazon announces or discloses anyways.

EagerDinosaur

From the accounts I've read, I'd say that the AWS internal processes were the main cause of the outage. Expecting engineers to always correctly type command line parameters on live systems isn't reasonable.

This hasn't discouraged me from using the cloud. The on-premises infrastructure at the large IT company I work for seems considerably less reliable than AWS.

DatabaseHead

My bad wrong link posted. I was following a twitter feed that mentioned he was let go, of course heresay so worthless nonetheless. Will be interesting to see what other news comes out in the next couple of days.

And any disciplinary measures certainly wouldn't be something that Amazon announces or discloses anyways.

Well of course not

, I was thinking along the lines of unnamed sources. But of course that would never happen.....