For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system.
At some point you need to start worrying about asteroid defense too.
Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?
The footnote which says that is the design durability against equipment failure literally begins:
In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
In US courts you can sue anyone for anything but you might not win. The US government has sovereign immunity from most civil liability. As for the legal system in Bahrain I have no idea but hypothetically even if Amazon could somehow win a judgment they wouldn't be able to collect.
Let's just revisit all the different acts of war that the EU has already done:
* Seizing sovereign reserves
* Seizing Russian ships on the high seas
* Sending special forces and other "military advisers" to launch long range attacks on Russian soil.
* Sending in commandos to launch attacks on nuclear power plants
You really don't think this will lead to war? I think there is a 50% chance there will be open war, and that percentage increases as the EU continues to take these measures, each of which are equivalent to acts of war from the perspective of international law.
And when this happens, and EU data centers are hit, you will probably still blame America and then not make any connection with data residency requirements and the resulting chaos.
Putin is launching and provoking wars, not the EU.
you will probably still blame America
No, we would blame Putin, because he's the blameworthy party.
As for Iran, Trump (and Trump voters by extension) is the blameworthy party. I don't even think any other R would have been dumb enough to go at Iran like this.
The EU committed all of these acts, each of which is considered an act of war. The fact that you think Russia deserves it makes no difference whatsoever to the likelihood of it causing a war.
You are really obsessed with blaming groups of people, which is a terrible reason to commit acts of war. When you go up to someone and punch them in the face, the fact that you think they deserve it has zero influence on whether they decide to punch you back. You need to understand that instead of viewing the world as an emotionally incontinent child. This is for grownups, not morally outraged children, and the EU needs to grow up, and not do something because they think the other side deserves it, but to consider how the other side will react.
Now if Russia launches long range strikes against EU infrastructure, and seizes EU ships, and sends commando raids into EU territory, you will be stunned, stunned and will mutter to yourself "How could this possibly happen?? How dare they attack me for no reason?!" and this is how you blunder into a war that you will certainly lose, all because you did not think of how the other side will react, and instead focused on your own misguided blaming of others, thinking your own moral outrage justifies seizing sovereign assets, seizing ships, and shooting missiles deep into the territory of your opponent, thinking there will be no repercussions because in your own mind, they deserve it.
Poor Russia, forced to invade Ukraine! My heart bleeds for the Russian army, so cruelly dragged across the borders into another country and forced to fight its way out.
Poor defenseless Ukraine, ethnically cleansing their Russian speaking population and then being shocked, shocked, that they are invaded. Those banderists should be free to commit all the atrocities they want within their own borders and to violate all the Minsk agreements guaranteeing sovereignty to their ethnic Russian populations. How shocking and surprising that they got into a war.
Just as shocking for when war comes to Europe, you will be stunned that this happened for no reason. You will be completely flabbergasted. All of history will unfold, taking you completely by surprise if you are unable to link cause to effect, and believe that your moral condemnation is all that needs to be agreed upon before launching attacks at a foreign nation.
Just as the US was shocked, stunned, that Iran closed the straight of Hormuz for no reason and shot missiles at US military bases in an act of unprovoked aggression. Isn't it enough to be morally outraged? Isn't that a good enough reason to attack a nation? What kind of world are we living in when being morally correct doesn't shield you from the other side punching back?
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
In a war where schools and bridges and aid convoys and bread lines are targeted
Yes but that's clearly not this war. This is a missile war where high-value strategic military targets are the priority. The US and Israel hit some prisons, damaged some hospitals, and ofc killed 168 schoolchildren. They also targeted residential areas to assassinate important leaders. But Iran has not returned that. They've stuck pretty strictly to military targets with very few exceptions
If this was a war where bridges were a target, Iran would not be so successful. There's simply a too limited amount of missiles. Also only 20% of Saudi Arabia has citizenship while almost the whole rest is basically indentured servants. It's not like they could provoke a popular uprising or anything. There's no strategic value in hitting "bread lines"
An American-run data center is much more likely to be targeted than some locally-owned and run company that isn't obviously connected to a foriegn power or the state that hosts it.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
How about ... stop bombing other countries? Trump is like a
professional liar. From "no more forever wars" to "hey this is
what must be done now" in a second. He is almost as good as
Putin with regards to lies - the ultimate agent Krasnov. Minus
the apparent dementia now.
You can't have a bunch of deranged mullahs threaten the entire region and world with missiles or nukes. Chanting "death to America" for half a century and killing thousands of Americans directly or indirectly. America should have blasted the hell out of the mullahs when they took hostages about 50 years ago. This is the first administration in a long time with the cohones to do something about it.
This is not exactly a nuanced view of the conflict, and in either case, the fact that you don't like that someone on the other side of the world is chanting death to America doesn't give you a bonus card for a free attack.
Seriously, it's like people, when deciding whether to launch a war or not, are not thinking "how will the other side react and will this conflict benefit me" but instead they are only thinking "does this nation deserve to get hit".
Well, news flash, your moral outrage does not translate into you not suffering more than your opponent during a conflict. It's a completely separate issue, and a personal issue between you and your priest or rabbi. When it comes to starting wars, you have to look at military capabilities and long term outcomes, not "does this nation deserve to be attacked".
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
In the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done.
If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?
Obviously what you understand is different from the reality after you actually follow all the footnotes.
Did they say its S3 data? Could also be single-az EBS or RDS.
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
[1] https://news.ycombinator.com/item?id=49033240
[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
https://aws.amazon.com/compliance/data-privacy-faq/
"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
That would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.
This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
AZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region.
Regions were always the scale of disaster isolation on AWS.
Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.
I think so.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
They guaranteed 11 9's durability, didn't they?
e: Yep
https://aws.amazon.com/s3/storage-classes/
I don't think this has any teeth. They don't compensate in the event of loss afaict.
Considering all the data they have globally, they might still be compliant.
Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.
They say it's "designed for" 11 9s, not guaranteed.
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.
The SLA excludes force majeure.
This
Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.
Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?
The footnote which says that is the design durability against equipment failure literally begins:
I'm sorry but your data is in another castle.
This should have been in that super Dario game.
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
For long-term backups you want offline cold storage in an underground facility in a friendly jurisdiction, not a datacenter.
War did not randomly came. America intentionally caused it.
And has lawless goverment and unaccountable tech industry making it bad place for data.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
In US courts you can sue anyone for anything but you might not win. The US government has sovereign immunity from most civil liability. As for the legal system in Bahrain I have no idea but hypothetically even if Amazon could somehow win a judgment they wouldn't be able to collect.
You don't see the EU causing a war with Russia?
Let's just revisit all the different acts of war that the EU has already done:
You really don't think this will lead to war? I think there is a 50% chance there will be open war, and that percentage increases as the EU continues to take these measures, each of which are equivalent to acts of war from the perspective of international law.
And when this happens, and EU data centers are hit, you will probably still blame America and then not make any connection with data residency requirements and the resulting chaos.
Putin is launching and provoking wars, not the EU.
No, we would blame Putin, because he's the blameworthy party.
As for Iran, Trump (and Trump voters by extension) is the blameworthy party. I don't even think any other R would have been dumb enough to go at Iran like this.
The EU committed all of these acts, each of which is considered an act of war. The fact that you think Russia deserves it makes no difference whatsoever to the likelihood of it causing a war.
You are really obsessed with blaming groups of people, which is a terrible reason to commit acts of war. When you go up to someone and punch them in the face, the fact that you think they deserve it has zero influence on whether they decide to punch you back. You need to understand that instead of viewing the world as an emotionally incontinent child. This is for grownups, not morally outraged children, and the EU needs to grow up, and not do something because they think the other side deserves it, but to consider how the other side will react.
Now if Russia launches long range strikes against EU infrastructure, and seizes EU ships, and sends commando raids into EU territory, you will be stunned, stunned and will mutter to yourself "How could this possibly happen?? How dare they attack me for no reason?!" and this is how you blunder into a war that you will certainly lose, all because you did not think of how the other side will react, and instead focused on your own misguided blaming of others, thinking your own moral outrage justifies seizing sovereign assets, seizing ships, and shooting missiles deep into the territory of your opponent, thinking there will be no repercussions because in your own mind, they deserve it.
Poor Russia, forced to invade Ukraine! My heart bleeds for the Russian army, so cruelly dragged across the borders into another country and forced to fight its way out.
Poor defenseless Ukraine, ethnically cleansing their Russian speaking population and then being shocked, shocked, that they are invaded. Those banderists should be free to commit all the atrocities they want within their own borders and to violate all the Minsk agreements guaranteeing sovereignty to their ethnic Russian populations. How shocking and surprising that they got into a war.
Just as shocking for when war comes to Europe, you will be stunned that this happened for no reason. You will be completely flabbergasted. All of history will unfold, taking you completely by surprise if you are unable to link cause to effect, and believe that your moral condemnation is all that needs to be agreed upon before launching attacks at a foreign nation.
Just as the US was shocked, stunned, that Iran closed the straight of Hormuz for no reason and shot missiles at US military bases in an act of unprovoked aggression. Isn't it enough to be morally outraged? Isn't that a good enough reason to attack a nation? What kind of world are we living in when being morally correct doesn't shield you from the other side punching back?
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
I don't think this is a flipside, unless you're thinking from the PoV of data itself rather than its owners.
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
https://www.edpb.europa.eu/system/files/documents/2021-06/ed...
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
"Use the cloud" they say.
"It's the only way." the true believers say.
"You can't run your own computer systems.", they say.
"Trust us.", they say.
“No one ever got fired for buying IBM.”
Wildberries has nothing to do with AWS.
"Massive centralisation of computing can never go wrong."
"We don't need to host our own systems, it's all safe in one of 20 global data centers."
etc etc
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
HN shitposting is the point. If you want actual pragmatic commentary, use Lobsters.
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
In a war where schools and bridges and aid convoys and bread lines are targeted, anyone should consider themselves a valid target.
It's true that you are less likely to be a target than a state-sponsored tech megacorp's data centers, but the risk is still present.
Risk is always present, We quantify it for a reason and then we mitigate if it's beyond a level we are comfortable.
Living is risk, every time I go to the shop for milk there is a non-zero chance I don't come back but the risk is so low I don't worry about it.
Yes but that's clearly not this war. This is a missile war where high-value strategic military targets are the priority. The US and Israel hit some prisons, damaged some hospitals, and ofc killed 168 schoolchildren. They also targeted residential areas to assassinate important leaders. But Iran has not returned that. They've stuck pretty strictly to military targets with very few exceptions
If this was a war where bridges were a target, Iran would not be so successful. There's simply a too limited amount of missiles. Also only 20% of Saudi Arabia has citizenship while almost the whole rest is basically indentured servants. It's not like they could provoke a popular uprising or anything. There's no strategic value in hitting "bread lines"
An American-run data center is much more likely to be targeted than some locally-owned and run company that isn't obviously connected to a foriegn power or the state that hosts it.
Not the OP, but the point is that decentralized infrastructure is a lot more resilient to attacks of any kind.
this is reductionist to the point of absurdism.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
Can you post the specs on your missile defense system for your home lab?
How are you dealing with the rise of low-cast swarm attacks from drones?
Is it land-based, sea-based or space-based and at what point of the trajectory do you target and do you use jamming?
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
"Daddy, where were you when the flames reached our house?"
"I was in the office, reviewing Terraform plans"
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
That's probably already true for 7 days or less
7 days of unreliable electricity wouldn't be unheard of for a very large storm
Did local laws permit?
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
That’s simply not true. I use S3 (well GCS mostly) for data that I wouldn’t be upset if it’s lost. And I pay the zonal discount for it.
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
Nothing is ever real-time. Eventual consistency leads to some data are not backed up.
You talking as if this is some mom-and-pop shop that you run.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries
Typically 300 miles geographically but could be hard in some Gulf States
In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all.
Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.
A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
Not all data is allowed to leave all countries.
What a nightmare scenario to tabletop. How do you even begin to recover from something like this?
Backups in a different region?
Works unless local laws specifically block you doing that which they do for some classes of data in some countries.
Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.
Do cloud providers even share data center locations so you can assess the "far enough" bit yourself?
No - and usually the reason is so they cannot be targeted.
Well… they have a good excuse.
How about ... stop bombing other countries? Trump is like a professional liar. From "no more forever wars" to "hey this is what must be done now" in a second. He is almost as good as Putin with regards to lies - the ultimate agent Krasnov. Minus the apparent dementia now.
It's a war that he started to benefit Israel, and is now causing suffering for so many others.
You can't have a bunch of deranged mullahs threaten the entire region and world with missiles or nukes. Chanting "death to America" for half a century and killing thousands of Americans directly or indirectly. America should have blasted the hell out of the mullahs when they took hostages about 50 years ago. This is the first administration in a long time with the cohones to do something about it.
This is not exactly a nuanced view of the conflict, and in either case, the fact that you don't like that someone on the other side of the world is chanting death to America doesn't give you a bonus card for a free attack.
Seriously, it's like people, when deciding whether to launch a war or not, are not thinking "how will the other side react and will this conflict benefit me" but instead they are only thinking "does this nation deserve to get hit".
Well, news flash, your moral outrage does not translate into you not suffering more than your opponent during a conflict. It's a completely separate issue, and a personal issue between you and your priest or rabbi. When it comes to starting wars, you have to look at military capabilities and long term outcomes, not "does this nation deserve to be attacked".
Backup both locally and everywhere no matter what.
50%+ of companies that lose all of their data go out of business in 6 months.
DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.
And a friendly reminder that replication isn't a tested data backup.
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.
turns out reading comprehension skills have still not gotten better in the future
In the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done.
If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
MENA is the on-prem capital of the world, don't worry.
Are you me from the future? I'm not talking to future strangers. Tell me to deliver the bad news myself from the future, in the future.
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
He forgot to add "if it's Multi-AZ" ;)
multi-AZ doesn’t help against multi-AZ drones :)
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
[delayed]