For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
We received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it.
nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office
Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system.
At some point you need to start worrying about asteroid defense too.
Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?
I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.
The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.
You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur.
Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.
Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.
Somehow I feel like the biggest post-apocalyptic problem will be the loss of home equity due to uninsured damage causing a collapse of financial markets.
The sales pitch should change from "probabilistically we will NEVER lose your data" to "you are most likely to lose your data due to wars, terrorists, software bugs, someone losing the master encryption key, the government forcing us to...".
Offsite backups are sadly rare these days, and aws sales is the main reason why.
Creative accounting works and is good because it works. If your customers give you more money because you lied to them, but it's legal, then it's good.
The footnote which says that is the design durability against equipment failure literally begins:
In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)
We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
In US courts you can sue anyone for anything but you might not win. The US government has sovereign immunity from most civil liability. As for the legal system in Bahrain I have no idea but hypothetically even if Amazon could somehow win a judgment they wouldn't be able to collect.
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
I'm comfortable enough with the Sydney and Melbourne AWS regions - about 700km (400 miles) apart and with (at least) 3 AZs in each. If something takes out enough AWS datacenters to lose some of work's or client data stored across all that, the uptime and resilience of the CRUD platforms I'm responsible for will not be very high on my personal priority list. (At least on AWS datacenter is within 10km of my home. I'm hoping that well before Australia gets involved in the sort of geopolitical conflict that might mean missile strikes against civilian infrastructure, I'll have headed bush to hang out with my off grid friends)
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
In a war where schools and bridges and aid convoys and bread lines are targeted
Yes but that's clearly not this war. This is a missile war where high-value strategic military targets are the priority. The US and Israel hit some prisons, damaged some hospitals, and ofc killed 168 schoolchildren. They also targeted residential areas to assassinate important leaders. But Iran has not returned that. They've stuck pretty strictly to military targets with very few exceptions
If this was a war where bridges were a target, Iran would not be so successful. There's simply a too limited amount of missiles. Also only 20% of Saudi Arabia has citizenship while almost the whole rest is basically indentured servants. It's not like they could provoke a popular uprising or anything. There's no strategic value in hitting "bread lines"
An American-run data center is much more likely to be targeted than some locally-owned and run company that isn't obviously connected to a foriegn power or the state that hosts it.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult.
Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th.
I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough.
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
Google internally has lots of possible redundancy levels for data.
They don't sell any of the lower and less reliable levels to the public, I suspect simply because the reputational damage from losing user data is so bad, and the news will take no notice of the fact the user got a discount for less reliable storage.
Most of Google's customers wouldn't know how to choose anyways, if these were exposed. My memory was quite hazy but I recall having a discussion with my colleague on choosing which Reed–Solomon code for our Colossus files, and apparently the choice was down to RS(8,3) or RS(9,3). I don't think even as Googlers we really had enough information to make an informed choice. Comparatively it was much easier to decide which cells to use for multi-location replication in Placer.
It's a great service for large caches. For example, we process a lot of imagery that we download from third-party providers. We save a lot of latency by storing the data in a single-AZ S3.
If it dies, we will just have to re-download the data.
EBS data data (sic) is unrecoverable, and that's what's reported.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
The article does not tell us what products were impacted.
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
They're likely going to target two datacenters, not the datacenters + your medium sized company's office NAS and the safe in the office manager's home.
Yes and no. For example if you are doing Azure, technically Azure can see tenant traffic I believe and you need to use both a Platform Key and CMK for data rest. VMs need encryption at host turned on too.
There is nothing to say that a determined adversary may still get at your data so it needs to stay in country.
Yeah same with AWS - they say they can't see my custom KMS key but... this stuff all lives on their servers, not on my servers. AWS definitely have the ability to see my KMS keys and decrypt my data but I assume that they won't unless a judge tells them to.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.
[1] That's a quote, including the Boston accent.
[2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
You usually get city level location information. Depends on your definition for 'far enough' if that works for you.
me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough.
250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.
I wonder what exactly these laws prohibit. Like, does it apply to a fully encrypted cold-copy on Amazon Glacier? If you don't store the encryption key outside of UAE, I'd say this isn't even the same data that gets transferred to the third party, it's just some random blob. But I have no idea if the authorities of that country would agree and if it's even actually enforced for that matter, or if it's one of those laws that actually cause problems only if you follow them.
For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably
a) letting customers in other areas know that their data is backed up to another continent
b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"
For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
In the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done.
If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop insisting on where the data resided for jurisdictional purposes and start talking about where administrative access and encryption keys lived.
[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!
and start talking about where administrative access and encryption keys lived
Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.
I'm disagreeing with you. I in the before time, I had all sorts of conversations around this topic with any number of cloud providers that were like:
Us: We are concerned about our citizens (US) data, how are you managing the databases.
Clout Provider (CP): They are only managed by fully background check employees.
Us: Yeah, but where are they? What is their citizenship?
CP: Um...mostly Eastern Europe. Lots in RU. (another CP proudly said "they're pretty much all in China...for cost containment").
Us: ...
Us: We are concerned about our citizens (EU) data, how are you managing encryption?
CP: Everything is perfectly encrypted with hardware HSMs and all the FIPS and stuff.
Us: So...where are the folks who run the HSMs?
CP: Um...mostly SV. Some in the EU.
Us: But can you assemble a quorum of US citizens for the HSM?
CP: Of course!
Us: ...
And on and on. Not to put too fine a point on it, many of us have no faith that vendors self policing international data protection in the face of government level pressure on companies and employees would work. Not that it can't, I don't think it would.
(I like the accidental pun of "Clout Provider" btw, which sadly conveys some of what they try to imply).
We may not be disagreeing that much. My argument was, and is, it's not about where the data is, it's about who has control over it. The counter-argument was "well if it's in another country, then we don't have jurisdiction, so it's going to be much harder". But what you need jurisdiction over is the people. Otherwise, you end up with multi-national corporate end-runs where you have shonky companies offering to store data locally, but who knows what department has control and access.
To be fair, the context I was having these conversations was countries arguing for data residency to combat the threat of mass surveillance (corporate and governmental) in the US, and the limited protections their users had relative to US nationals. But again, the problem is that it assumes that jurisdiction remains territorial: which is not how this was ever going to play out. The next wave after data residency requirements, beyond the usual extraterritorial intelligence community actions, was laws like the US CLOUD Act, the UK's Investigatory Powers Act, and Australia's TIA law, which effectively attempts to provide regular government departments and law enforcement with the legal ability to access data that would technically be on foreign soil.
My point was not that corporations should not self-police, but the concept of "it's stored here so we can oversee it" is not as clearcut as it seemed, and it risks introducing a new level of complexity to resiliently storing data. Which may be worth the price, but was never considered at the level this was discussed.
That's fair, and it sounds like we aren't that far apart. It is, in fact, about control. So I'll restate my central theme as "until the idea of enforceable data sovereignty requirements were enshrined in law, the cloud providers did not and would not delegate control of any body of data to 'controllers' that weren't in jurisdictions where they could be influenced/coerced to compromise that data". Was this a slippery slope/camel in the tent? Well...that's politics and it didn't have to be, but I see your point. But the reality is the push for data sovereignty wasn't done with the intention of enabling totalitarian follow-on legislation and it wasn't in and of itself a bad idea.
Yep, exactly. There's a peculiarly unsatisfying kind of vindication that comes from making "slippery slope" arguments, and then watch them play, and now you are now both a) technically correct, and b) fucked. You'll excuse me if I have a brief "I told you so" moment about a scenario about Amazon's UAE datacenters being bombed without bakcups because of a US-instigated Iranian conflict, where -- if I'd ever dared to describe it -- would definitely have got me laughed out of those rooms in 2010.
I think both of these things are (very) often true, but it is also true that if I'm going to have backups, it is (all other things being equal) better to minimize correlated risk. The assumption in a lot of these conversations is that having the data "in one place" (ie inside a country) was "safer" than having it in "somewhere else". The tougher counterintuitive argument was that it can be safer to have data stored in multiple places, for some risk assessments -- and that for others, having data close by was less important, in the case of seizure or surveillance or illegal use, than who had legal or effective access to that data.
This is a very engineer-centric view. I studied economics in school, so an analogy in that realm is ironically how all countries should specialize and raise the PPC curve. The reality of the situation was that in 2010 not many people understood how powerful big data actually was. Data sovereignty is actually quite logical when you consider the scale and power of not only the company, but the US as a whole. I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws. You can also create reasonably diversified data silos within a country.
As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.
See my other answers, but briefly: no, they were not thinking about this, which is why we were raising it. I guess the counterintuitive point we were trying to get across is that with most things, the best way to keep it safe from being lost is to put it in a known place, and lock it away. But for data, the strategy -- for that scenario -- is to keep it in a lot of places, with heterogenous defense strategies. This is for data loss, of course, not access or surveillance or unlawful processing. But there is a cost as well as a benefit to deliberately limiting your options.
(I can feel someone saying "but surely having redundancy in one country is good enough, so I'll just say that I know relatively sane people who try to have hemispheric redundancy in their data, and also you never know when two different-in-every-quality-but one locations will suffer from the same disaster. Floods; heat-waves; national protests and strikes. It's surprising how often rare things happen!)
On your second point, it really is crazy. And also amazing that this is a capability that is -- or should be -- available to anyone in the world, not just in the US, and not just devs. Hopefully without also having to think about their data suddenly finding itself in a warzone.
This is for data loss, of course, not access or surveillance or unlawful processing.
This is why the minority of politicians who actually know about how this stuff works worry about where the data resides for jurisdictional purposes. If the government where the data resides can compel the folks who have physical and/or logical access to the physical machines that contain that data to give them access to that data, then that's game over for you.
«But you just don't permit that sort of breach to happen!» you might say. To which I reply "Yeah, right.".
Substantial physical separation of datacenters is very important, but the politics and policies of the location housing the data cannot be ignored.
You can also create reasonably diversified data silos within a country.
I generally agree, although if a small country only had half a dozen or so redundant data centers then it would be relatively easy for a powerful adversary to wipe out all of the data centers and potentially have a significant economic impact on that country.
Having a backup data center in an ally country might make sense. Kind of like how I keep an encrypted backup hard drive at my parents house. Whenever I go to visit I pull it out and backup my laptop there too.
... in the UAE, so only less risky if your office is remote enough it doesn't also make a nice target (like, say, located within/near Dubai's Internet City).
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
That's absolutely something you can configure your S3 bucket to do if you want (I have one of mine replicating elsewhere).
The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs.
S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.
The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive
It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate.
So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so.
And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice.
The minute you cross borders you run into data sovereignty concerns. No idea if it applies at the subnational level (US states, Emirates in the UAE etc) but it wouldn't surprise me if it did in some cases.
"The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand," AWS said in the status update
That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.
If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying.
I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.
I think a lot of people have built those services. But doing it right is more expensive, and people end up choosing the $5-$10/mo option over your $30/mo+ that does it right. Multiply those numbers by whatever multiple you want for higher end stuff.
Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy. And in the US at least, unhappy shareholders means lawsuits since publicly traded companies are legally bound to return growth in share price or dividends.
What?! Publicly traded companies are not “legally bound to return growth in share price or dividends.”
There are plenty of companies who pay no dividends and have not returned growth in share price. They’re still operating and no one’s coming to throw the execs in jail.
On the off chance that there is such a law, please cite it.
Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy.
This is true. The latter half of your comment is not. At best they have a duty to shareholders. But your assertion would mean every time a company posted a loss and the price went down the execs would be in legal trouble, which is nonsense.
Shareholder nuisance lawsuits can happen basically any time the stock goes down... or doesn't go up enough, and often companies will settle instead of the expensive fighting...
Yep, a lot of non-technical leaders believe that 'cloud' is synonymous with 'DR strategy' or even 'backup'. "We won't have to worry about being offline if our server goes down if we move to the cloud!" Some of these people fundamentally don't understand what the cloud is, their assumption is cloud means easy button that solves all your infrastructure and uptime problems.
And for a time, it even seems true as long as the natural disasters that impact various customer offices happen to not impact that one particular data center…
No, that quotation on the GP clearly states that AWS has enough redundancy within the same region that they will continue all services running on it if a datacenter is destroyed.
It's very clearly not about you being able to set-up redundancy for yourself.
That's actually true. AWS is designed to survive one datacenter being offline (which happened more than once, btw). When the first DC in ME was hit, AWS continued working normally, with only a few services experiencing issues.
But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.
Pretty much this, it’s your responsibility to use their tools to make sure your data is managed in such a way that any data destroyed is already elsewhere before the event.
The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.
If they're region-locking data appropriately (i.e. for people to comply with domestic storage / gdpr-style requirements) they really can't. Bahrain is only about 300 square miles / 80k hectares.
They could, of course, open other regional data centers in other countries, or say "data in this geo zone may be in any of X, Y or Z" countries, but for the latter that pretty starkly limits some of the major customers they'd have, I would guess, and for the former, well, they have other geo zones already, so if people weren't replicating to them, I'm not sure why adding me-east-1 me-west-1 me-central-1 would fix that issue, they just wouldn't replicate there either.
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".
It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.
The wording certainly seems very careful:
"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat)
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat)
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
The data center layout should look something like this:
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat) ???
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.
If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.
11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.
Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?
Obviously what you understand is different from the reality after you actually follow all the footnotes.
Did they say its S3 data? Could also be single-az EBS or RDS.
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
[1] https://news.ycombinator.com/item?id=49033240
[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
https://aws.amazon.com/compliance/data-privacy-faq/
"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
That would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.
This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
We received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it.
nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office
AZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region.
Regions were always the scale of disaster isolation on AWS.
Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.
I think so.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
They guaranteed 11 9's durability, didn't they?
e: Yep
https://aws.amazon.com/s3/storage-classes/
I don't think this has any teeth. They don't compensate in the event of loss afaict.
Considering all the data they have globally, they might still be compliant.
Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.
But do they have Act of Special Operation exemptions?
They say it's "designed for" 11 9s, not guaranteed.
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.
The SLA excludes force majeure.
This
Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.
Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?
I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.
The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.
Cosmic rays and dying hard drives are not force majeure though.
You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur.
Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.
Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.
Force majeure carveouts are really common in every type of contract.
You should check your home insurance contract, for instance... It likely would not cover an ICBM strike.
Somehow I feel like the biggest post-apocalyptic problem will be the loss of home equity due to uninsured damage causing a collapse of financial markets.
I think it's what most people comparing provider SLAs would expect
Are you suggesting that their technical documents have separate availability numbers to predict geopolitical events and war?
The sales pitch should change from "probabilistically we will NEVER lose your data" to "you are most likely to lose your data due to wars, terrorists, software bugs, someone losing the master encryption key, the government forcing us to...".
Offsite backups are sadly rare these days, and aws sales is the main reason why.
Creative accounting works and is good because it works. If your customers give you more money because you lied to them, but it's legal, then it's good.
The footnote which says that is the design durability against equipment failure literally begins:
It seems unreasonable to blame Amazon here. The AZ was destroyed. Are they supposed to have missile/drone defense?
I'm not going to complain to DoorDash if my order is delayed due to a car crash
I'm sorry but your data is in another castle.
This should have been in that super Dario game.
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
For long-term backups you want offline cold storage in an underground facility in a friendly jurisdiction, not a datacenter.
We've beer-o-clock wargamed this a bit.
If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)
We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
More likely than the network going down is you getting banned from the network because someone thought you were selling porn.
War did not randomly came. America intentionally caused it.
And has lawless goverment and unaccountable tech industry making it bad place for data.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
In US courts you can sue anyone for anything but you might not win. The US government has sovereign immunity from most civil liability. As for the legal system in Bahrain I have no idea but hypothetically even if Amazon could somehow win a judgment they wouldn't be able to collect.
Why wouldn't they be able to collect? As long as Amazon does business in a country the legal system in that country can collect.
How would the Bahrain legal system force the USA to pay reparations for the war it started? Seize all US assets, the way we did with Russia?
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
Ah yeah, the EU, a historically stable area.
Border between France and Germany is a particularly well know peaceful area, especially Alsace-Lorraine.
I don't think this is a flipside, unless you're thinking from the PoV of data itself rather than its owners.
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
https://www.edpb.europa.eu/system/files/documents/2021-06/ed...
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
I'm comfortable enough with the Sydney and Melbourne AWS regions - about 700km (400 miles) apart and with (at least) 3 AZs in each. If something takes out enough AWS datacenters to lose some of work's or client data stored across all that, the uptime and resilience of the CRUD platforms I'm responsible for will not be very high on my personal priority list. (At least on AWS datacenter is within 10km of my home. I'm hoping that well before Australia gets involved in the sort of geopolitical conflict that might mean missile strikes against civilian infrastructure, I'll have headed bush to hang out with my off grid friends)
"Use the cloud" they say.
"It's the only way." the true believers say.
"You can't run your own computer systems.", they say.
"Trust us.", they say.
“No one ever got fired for buying IBM.”
Wildberries has nothing to do with AWS.
"Massive centralisation of computing can never go wrong."
"We don't need to host our own systems, it's all safe in one of 20 global data centers."
etc etc
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
HN shitposting is the point. If you want actual pragmatic commentary, use Lobsters.
Can't, not invited
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
In a war where schools and bridges and aid convoys and bread lines are targeted, anyone should consider themselves a valid target.
It's true that you are less likely to be a target than a state-sponsored tech megacorp's data centers, but the risk is still present.
Risk is always present, We quantify it for a reason and then we mitigate if it's beyond a level we are comfortable.
Living is risk, every time I go to the shop for milk there is a non-zero chance I don't come back but the risk is so low I don't worry about it.
Yes but that's clearly not this war. This is a missile war where high-value strategic military targets are the priority. The US and Israel hit some prisons, damaged some hospitals, and ofc killed 168 schoolchildren. They also targeted residential areas to assassinate important leaders. But Iran has not returned that. They've stuck pretty strictly to military targets with very few exceptions
If this was a war where bridges were a target, Iran would not be so successful. There's simply a too limited amount of missiles. Also only 20% of Saudi Arabia has citizenship while almost the whole rest is basically indentured servants. It's not like they could provoke a popular uprising or anything. There's no strategic value in hitting "bread lines"
An American-run data center is much more likely to be targeted than some locally-owned and run company that isn't obviously connected to a foriegn power or the state that hosts it.
Not the OP, but the point is that decentralized infrastructure is a lot more resilient to attacks of any kind.
this is reductionist to the point of absurdism.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
Can you post the specs on your missile defense system for your home lab?
How are you dealing with the rise of low-cast swarm attacks from drones?
Is it land-based, sea-based or space-based and at what point of the trajectory do you target and do you use jamming?
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
I would but then I'd have to kill you.
No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
"Daddy, where were you when the flames reached our house?"
"I was in the office, reviewing Terraform plans"
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
That's probably already true for 7 days or less
7 days of unreliable electricity wouldn't be unheard of for a very large storm
Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult.
Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th.
I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough.
[1] https://web.archive.org/web/20061206181753/http://interdicto...
You also need to understand whether the generator backup actually runs everything. Where I work it doesn't. Only "essential" systems get backup power.
And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.
Did local laws permit?
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
That’s simply not true. I use S3 (well GCS mostly) for data that I wouldn’t be upset if it’s lost. And I pay the zonal discount for it.
Google internally has lots of possible redundancy levels for data.
They don't sell any of the lower and less reliable levels to the public, I suspect simply because the reputational damage from losing user data is so bad, and the news will take no notice of the fact the user got a discount for less reliable storage.
Most of Google's customers wouldn't know how to choose anyways, if these were exposed. My memory was quite hazy but I recall having a discussion with my colleague on choosing which Reed–Solomon code for our Colossus files, and apparently the choice was down to RS(8,3) or RS(9,3). I don't think even as Googlers we really had enough information to make an informed choice. Comparatively it was much easier to decide which cells to use for multi-location replication in Placer.
You call it "discount" but it's a 20% discount on a 10x inflated price, so it's an 8x inflated price
Single AZ S3 has other benefits. The point isn't the price, it's that it's _highly performant_ since you can keep all of your reads in the same AZ
Yeah agreed - any ephemeral stuff I need is generally in DynamoDB - S3 (and database) are for permanent storage.
It's a great service for large caches. For example, we process a lot of imagery that we download from third-party providers. We save a lot of latency by storing the data in a single-AZ S3.
If it dies, we will just have to re-download the data.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
Nothing is ever real-time. Eventual consistency leads to some data are not backed up.
You talking as if this is some mom-and-pop shop that you run.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries
Typically 300 miles geographically but could be hard in some Gulf States
In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all.
Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.
They're likely going to target two datacenters, not the datacenters + your medium sized company's office NAS and the safe in the office manager's home.
(Encryption handles confidentiality concerns.)
Which is why data residency is such a stupid concept.
Yes and no. For example if you are doing Azure, technically Azure can see tenant traffic I believe and you need to use both a Platform Key and CMK for data rest. VMs need encryption at host turned on too.
There is nothing to say that a determined adversary may still get at your data so it needs to stay in country.
Yeah same with AWS - they say they can't see my custom KMS key but... this stuff all lives on their servers, not on my servers. AWS definitely have the ability to see my KMS keys and decrypt my data but I assume that they won't unless a judge tells them to.
A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.
Do you think they could ask for a backup
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
Isn't S3 claiming eleven nines of data durability?
https://aws.amazon.com/s3/storage-classes/
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
1/f noise strikes again
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
Not all data is allowed to leave all countries.
what a silly notion.
You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.
[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
What a nightmare scenario to tabletop. How do you even begin to recover from something like this?
Backups in a different region?
Works unless local laws specifically block you doing that which they do for some classes of data in some countries.
Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.
Do cloud providers even share data center locations so you can assess the "far enough" bit yourself?
No - and usually the reason is so they cannot be targeted.
You usually get city level location information. Depends on your definition for 'far enough' if that works for you.
me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough.
250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.
I wonder what exactly these laws prohibit. Like, does it apply to a fully encrypted cold-copy on Amazon Glacier? If you don't store the encryption key outside of UAE, I'd say this isn't even the same data that gets transferred to the third party, it's just some random blob. But I have no idea if the authorities of that country would agree and if it's even actually enforced for that matter, or if it's one of those laws that actually cause problems only if you follow them.
For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably
a) letting customers in other areas know that their data is backed up to another continent
b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"
For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.
Well… they have a good excuse.
And witness
Backup both locally and everywhere no matter what.
50%+ of companies that lose all of their data go out of business in 6 months.
DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.
And a friendly reminder that replication isn't a tested data backup.
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.
turns out reading comprehension skills have still not gotten better in the future
In the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done.
If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
MENA is the on-prem capital of the world, don't worry.
... but not only in an Azure datacenter!
(The obvious option here might be an encrypted backup on some hard drives in a safe in a local office.)
Are you me from the future? I'm not talking to future strangers. Tell me to deliver the bad news myself from the future, in the future.
I'm from the past and I concur.
Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop insisting on where the data resided for jurisdictional purposes and start talking about where administrative access and encryption keys lived.
[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!
and start talking about where administrative access and encryption keys lived
Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.
I'm not sure you were disagreeing with (past) me; but if you were, could you expand on your point?
I'm disagreeing with you. I in the before time, I had all sorts of conversations around this topic with any number of cloud providers that were like:
Us: We are concerned about our citizens (US) data, how are you managing the databases. Clout Provider (CP): They are only managed by fully background check employees. Us: Yeah, but where are they? What is their citizenship? CP: Um...mostly Eastern Europe. Lots in RU. (another CP proudly said "they're pretty much all in China...for cost containment"). Us: ...
Us: We are concerned about our citizens (EU) data, how are you managing encryption? CP: Everything is perfectly encrypted with hardware HSMs and all the FIPS and stuff. Us: So...where are the folks who run the HSMs? CP: Um...mostly SV. Some in the EU. Us: But can you assemble a quorum of US citizens for the HSM? CP: Of course! Us: ...
And on and on. Not to put too fine a point on it, many of us have no faith that vendors self policing international data protection in the face of government level pressure on companies and employees would work. Not that it can't, I don't think it would.
(I like the accidental pun of "Clout Provider" btw, which sadly conveys some of what they try to imply).
We may not be disagreeing that much. My argument was, and is, it's not about where the data is, it's about who has control over it. The counter-argument was "well if it's in another country, then we don't have jurisdiction, so it's going to be much harder". But what you need jurisdiction over is the people. Otherwise, you end up with multi-national corporate end-runs where you have shonky companies offering to store data locally, but who knows what department has control and access.
To be fair, the context I was having these conversations was countries arguing for data residency to combat the threat of mass surveillance (corporate and governmental) in the US, and the limited protections their users had relative to US nationals. But again, the problem is that it assumes that jurisdiction remains territorial: which is not how this was ever going to play out. The next wave after data residency requirements, beyond the usual extraterritorial intelligence community actions, was laws like the US CLOUD Act, the UK's Investigatory Powers Act, and Australia's TIA law, which effectively attempts to provide regular government departments and law enforcement with the legal ability to access data that would technically be on foreign soil.
My point was not that corporations should not self-police, but the concept of "it's stored here so we can oversee it" is not as clearcut as it seemed, and it risks introducing a new level of complexity to resiliently storing data. Which may be worth the price, but was never considered at the level this was discussed.
That's fair, and it sounds like we aren't that far apart. It is, in fact, about control. So I'll restate my central theme as "until the idea of enforceable data sovereignty requirements were enshrined in law, the cloud providers did not and would not delegate control of any body of data to 'controllers' that weren't in jurisdictions where they could be influenced/coerced to compromise that data". Was this a slippery slope/camel in the tent? Well...that's politics and it didn't have to be, but I see your point. But the reality is the push for data sovereignty wasn't done with the intention of enabling totalitarian follow-on legislation and it wasn't in and of itself a bad idea.
Best laid plans and all that.
Yep, exactly. There's a peculiarly unsatisfying kind of vindication that comes from making "slippery slope" arguments, and then watch them play, and now you are now both a) technically correct, and b) fucked. You'll excuse me if I have a brief "I told you so" moment about a scenario about Amazon's UAE datacenters being bombed without bakcups because of a US-instigated Iranian conflict, where -- if I'd ever dared to describe it -- would definitely have got me laughed out of those rooms in 2010.
Is there not more than one data center in your country? Is the power feed at your office too small to put a computer there?
I think both of these things are (very) often true, but it is also true that if I'm going to have backups, it is (all other things being equal) better to minimize correlated risk. The assumption in a lot of these conversations is that having the data "in one place" (ie inside a country) was "safer" than having it in "somewhere else". The tougher counterintuitive argument was that it can be safer to have data stored in multiple places, for some risk assessments -- and that for others, having data close by was less important, in the case of seizure or surveillance or illegal use, than who had legal or effective access to that data.
This is a very engineer-centric view. I studied economics in school, so an analogy in that realm is ironically how all countries should specialize and raise the PPC curve. The reality of the situation was that in 2010 not many people understood how powerful big data actually was. Data sovereignty is actually quite logical when you consider the scale and power of not only the company, but the US as a whole. I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws. You can also create reasonably diversified data silos within a country.
As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.
See my other answers, but briefly: no, they were not thinking about this, which is why we were raising it. I guess the counterintuitive point we were trying to get across is that with most things, the best way to keep it safe from being lost is to put it in a known place, and lock it away. But for data, the strategy -- for that scenario -- is to keep it in a lot of places, with heterogenous defense strategies. This is for data loss, of course, not access or surveillance or unlawful processing. But there is a cost as well as a benefit to deliberately limiting your options.
(I can feel someone saying "but surely having redundancy in one country is good enough, so I'll just say that I know relatively sane people who try to have hemispheric redundancy in their data, and also you never know when two different-in-every-quality-but one locations will suffer from the same disaster. Floods; heat-waves; national protests and strikes. It's surprising how often rare things happen!)
On your second point, it really is crazy. And also amazing that this is a capability that is -- or should be -- available to anyone in the world, not just in the US, and not just devs. Hopefully without also having to think about their data suddenly finding itself in a warzone.
This is why the minority of politicians who actually know about how this stuff works worry about where the data resides for jurisdictional purposes. If the government where the data resides can compel the folks who have physical and/or logical access to the physical machines that contain that data to give them access to that data, then that's game over for you.
«But you just don't permit that sort of breach to happen!» you might say. To which I reply "Yeah, right.".
Substantial physical separation of datacenters is very important, but the politics and policies of the location housing the data cannot be ignored.
I generally agree, although if a small country only had half a dozen or so redundant data centers then it would be relatively easy for a powerful adversary to wipe out all of the data centers and potentially have a significant economic impact on that country.
Having a backup data center in an ally country might make sense. Kind of like how I keep an encrypted backup hard drive at my parents house. Whenever I go to visit I pull it out and backup my laptop there too.
You can always selfhost
... in the UAE, so only less risky if your office is remote enough it doesn't also make a nice target (like, say, located within/near Dubai's Internet City).
and you don't upset whichever prince decides to compete
Check if the requirement is "only in the uae" or has to have the primary copy in the UAE
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
Bezos hasn't lost anything from this. He's only gotten richer.
I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
He forgot to add "if it's Multi-AZ" ;)
multi-AZ doesn’t help against multi-AZ drones :)
black swan events.
this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone.
if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.
It's not really a black swan event just an unlikely one. If it were one it would not have been something people talked about before.
That's absolutely something you can configure your S3 bucket to do if you want (I have one of mine replicating elsewhere).
The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs.
S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.
It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate.
So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so.
And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice.
PS: hi!
The minute you cross borders you run into data sovereignty concerns. No idea if it applies at the subnational level (US states, Emirates in the UAE etc) but it wouldn't surprise me if it did in some cases.
It applies subnationally. Think about different gambling laws per US state, and if you’re running the infrastructure for that across state lines.
The me-south-1 (Bahrain) Region was three AZs: https://docs.aws.amazon.com/global-infrastructure/latest/reg...
That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.
If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying.
I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.
I think a lot of people have built those services. But doing it right is more expensive, and people end up choosing the $5-$10/mo option over your $30/mo+ that does it right. Multiply those numbers by whatever multiple you want for higher end stuff.
Not to be callous but even the 3-2-1 rule is pretty basic, the issue is that people don't apply it. But that's a hiring and strategy thing.
Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy. And in the US at least, unhappy shareholders means lawsuits since publicly traded companies are legally bound to return growth in share price or dividends.
What?! Publicly traded companies are not “legally bound to return growth in share price or dividends.”
There are plenty of companies who pay no dividends and have not returned growth in share price. They’re still operating and no one’s coming to throw the execs in jail.
On the off chance that there is such a law, please cite it.
Not a law perhaps, but an ingrained neoliberal philosophy that is pervasive in certain management cultures in the US definitely.
This is true. The latter half of your comment is not. At best they have a duty to shareholders. But your assertion would mean every time a company posted a loss and the price went down the execs would be in legal trouble, which is nonsense.
Shareholder nuisance lawsuits can happen basically any time the stock goes down... or doesn't go up enough, and often companies will settle instead of the expensive fighting...
Yep, a lot of non-technical leaders believe that 'cloud' is synonymous with 'DR strategy' or even 'backup'. "We won't have to worry about being offline if our server goes down if we move to the cloud!" Some of these people fundamentally don't understand what the cloud is, their assumption is cloud means easy button that solves all your infrastructure and uptime problems.
You are pointing the fingers at the wrong people. Cloud providers pushed this idea onto executives via their marketing and conferencing channels.
Not disimilar to how AI naratives are pushed on executives these last two years.
And for a time, it even seems true as long as the natural disasters that impact various customer offices happen to not impact that one particular data center…
No, that quotation on the GP clearly states that AWS has enough redundancy within the same region that they will continue all services running on it if a datacenter is destroyed.
It's very clearly not about you being able to set-up redundancy for yourself.
That's actually true. AWS is designed to survive one datacenter being offline (which happened more than once, btw). When the first DC in ME was hit, AWS continued working normally, with only a few services experiencing issues.
But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.
That's fair. The only reference was about one datacenter blowing up. You can't have unlimited redundancy.
Pretty much this, it’s your responsibility to use their tools to make sure your data is managed in such a way that any data destroyed is already elsewhere before the event.
The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.
Isn’t the problem that multiple datacenters in one zone were blown up?
They had nine years and more money than god to build redundancy in an unstable region.
If it's an unstable region, then it might not result in a good return, especially since a blown up data center is 100% loss.
They did build redundancy but most of it was bombed.
If they're region-locking data appropriately (i.e. for people to comply with domestic storage / gdpr-style requirements) they really can't. Bahrain is only about 300 square miles / 80k hectares.
They could, of course, open other regional data centers in other countries, or say "data in this geo zone may be in any of X, Y or Z" countries, but for the latter that pretty starkly limits some of the major customers they'd have, I would guess, and for the former, well, they have other geo zones already, so if people weren't replicating to them, I'm not sure why adding me-east-1 me-west-1 me-central-1 would fix that issue, they just wouldn't replicate there either.
Multiple zones impacted.
that quote is definitely making its way into a lawsuit
There's something reassuring in this for me.
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".
Uh.
Uh-oh.
It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.
The wording certainly seems very careful:
"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
Hey but that's exactly as per design. It is the customer's responsibility to store stuff elsewhere as DR backup, not AWS.
That's the excuse they'll say, yes, then we quote back to them "eleven nines" and they come up with an excuse for that too
Guess those multi-AZ promises have an asterisk when actual missiles are involved. Makes you double-check your own off-site backups.
"..even if something happens only once in a billion requests, that means it happens multiple times per day within S3."
but one in a trillion...
├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3
The data center layout should look something like this:
Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.
If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.
See also previous discussion: https://news.ycombinator.com/item?id=49033240
us-east-1 is finally looking pretty stable for once.
11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.
Seems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.
There are many regulations over there which prevent data from leaving the country.
Sometimes those rules can have serious consequences.
Since roughly zero of the articles I've seen link back to the source of their quotes, here it is: <https://status.aws.amazon.com/#multipleservices-me-south-1_1...>