Obviously what you understand is different from the reality after you actually follow all the footnotes.

Did they say its S3 data? Could also be single-az EBS or RDS.

https://aws.amazon.com/compliance/data-privacy-faq/

"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."

This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.

We received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it.

nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office

Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.

I think so.

Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.

I don't think this has any teeth. They don't compensate in the event of loss afaict.

Considering all the data they have globally, they might still be compliant.

I'm going to reword my Terms of Service this second to add "any military operation" next to "acts of war". But I'm sure we'll then have to demonstrate whether paramilitary are assimilated to the military.

US refuses payout for soldiers who died in Iran 'because it isn't a war'

In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.

https://news.ycombinator.com/item?id=49596955

Ah so that's why we need a lunar base. To uphold S3's 11 9s of availability

This

Cosmic rays and dying hard drives are not force majeure though.

You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur.

Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.

Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.

Somehow I feel like the biggest post-apocalyptic problem will be the loss of home equity due to uninsured damage causing a collapse of financial markets.

I think it's what most people comparing provider SLAs would expect

The sales pitch should change from "probabilistically we will NEVER lose your data" to "you are most likely to lose your data due to wars, terrorists, software bugs, someone losing the master encryption key, the government forcing us to...".

Offsite backups are sadly rare these days, and aws sales is the main reason why.

Creative accounting works and is good because it works. If your customers give you more money because you lied to them, but it's legal, then it's good.

Excluding war as force majeure in the Middle East is the same as excluding high tide as force majeure building a sandcastle at low tide.

The footnote which says that is the design durability against equipment failure literally begins:

In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss

Yeah I agree with you. If they made a specific promise to some unlikely case out of their control, then I would expect compensation.

Did Amazon make such a promise? They didn't as far as I know. My understanding is they provide specific guarantees like given an AZ outage, your data is still safe (provided you architect correctly).

Hot take: if you have a service that is 11 nines reliable, but there is an underlying component whose reliability is lower, cap the nines to that component.

If they haven't changed it recently, 11 nines is the durability target by design, but it is not set in any SLA. the S3 SLA is focused on availability.

This should have been in that super Dario game.

More likely than the network going down is you getting banned from the network because someone thought you were selling porn.

I mean that is how it works in all legal systems. If you refuse to pay your assets will be seized. Which is why they would almost certainly not refuse.

I'm not a lawyer, but this seems like a difficult case.

For one thing, regardless of what the US did, they did not launch the attack that damaged the datacenters. That attack was provoked by US actions, but the attack was an intentional and voluntary act by Iran/IRGC. I don't think any of the usual settings for liability for the acts of others really apply here, but I'm not a lawyer.

You'd also need to find a court of competent jurisdiction. The US is out because of soveriegn immunity. The Military Claims Act prohibits claims that "arise from action by an enemy or result directly or indirectly from an act of the armed forces of the United States in combat" [1]. The Federal Tort Claims Act bars "any claim arising in a foreign country" as well as "any claim arising out of the combatant activities of the military or naval forces, or the Coast Guard, during time of war" [2] and precedent recognize a state of war without a declaration.

The courts in the country where the data centers were attacked might not be interested in addressing conduct of the US that didn't happen in their country. The courts in Iran are likely to consider the attack on the data center a legitimate act of war and not a tort; anyway good luck collecting against the US with a judgement from an Iranian court.

Amazon might have a better time making a claim against Iran, in the court where the attacks took place, but good luck collecting a judgement. Maybe a case in the International Court of Justice, but those have to be submitted by a country and involved countries must consent.

[1] https://uscode.house.gov/view.xhtml?path=/prelim@title10/sub... chapter section 2734 (b) (3)

[2] see page 28 and 29 of https://www.congress.gov/crs_external_products/R/PDF/R45732/...

Border between France and Germany is a particularly well know peaceful area, especially Alsace-Lorraine.

I don't think this is a flipside, unless you're thinking from the PoV of data itself rather than its owners.

data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.

https://www.edpb.europa.eu/system/files/documents/2021-06/ed...

I'm comfortable enough with the Sydney and Melbourne AWS regions - about 700km (400 miles) apart and with (at least) 3 AZs in each. If something takes out enough AWS datacenters to lose some of work's or client data stored across all that, the uptime and resilience of the CRUD platforms I'm responsible for will not be very high on my personal priority list. (At least on AWS datacenter is within 10km of my home. I'm hoping that well before Australia gets involved in the sort of geopolitical conflict that might mean missile strikes against civilian infrastructure, I'll have headed bush to hang out with my off grid friends)

"Daddy, where were you when the flames reached our house?"

"I was in the office, reviewing Terraform plans"

I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.

He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.

7 days of unreliable electricity wouldn't be unheard of for a very large storm

I need to bring this out:

https://craphound.com/overclocked/Cory_Doctorow_-_Overclocke...

You also need to understand whether the generator backup actually runs everything. Where I work it doesn't. Only "essential" systems get backup power.

And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.

I had a similar discussion in the low 1990s at a large electronic manufacturer that had a pair of Unisys mainframes, one in São Paulo and another in Manaus, in the Amazon region communicating over a satellite link (two large 3m dishes on both places). There was one question of what happens if both fail at the same time, and I pointed that anything that takes out São Paulo and Manaus at the same time is a civilisation ending event, and we shouldn’t worry too much about that.

Anyway, we had a load shedding agreement with a large bank across the street from São Paulo and we could switch over to their Unisys mainframe in a matter of minutes, and vice versa.

Did local laws permit?

Most of Google's customers wouldn't know how to choose anyways, if these were exposed. My memory was quite hazy but I recall having a discussion with my colleague on choosing which Reed–Solomon code for our Colossus files, and apparently the choice was down to RS(8,3) or RS(9,3). I don't think even as Googlers we really had enough information to make an informed choice. Comparatively it was much easier to decide which cells to use for multi-location replication in Placer.

You call it "discount" but it's a 20% discount on a 10x inflated price, so it's an 8x inflated price

Single AZ S3 has other benefits. The point isn't the price, it's that it's _highly performant_ since you can keep all of your reads in the same AZ

Yeah agreed - any ephemeral stuff I need is generally in DynamoDB - S3 (and database) are for permanent storage.

It's a great service for large caches. For example, we process a lot of imagery that we download from third-party providers. We save a lot of latency by storing the data in a single-AZ S3.

If it dies, we will just have to re-download the data.

EBS data data (sic) is unrecoverable, and that's what's reported.

I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."

The article does not tell us what products were impacted.

… right … which conflicts with EBS being what's reported …

(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)

But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.

Nothing is ever real-time. Eventual consistency leads to some data are not backed up.

You talking as if this is some mom-and-pop shop that you run.

That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.

Do you think they could ask for a backup

1/f noise strikes again

If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.

Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.

what a silly notion.

You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.

[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.

No - and usually the reason is so they cannot be targeted.

You usually get city level location information. Depends on your definition for 'far enough' if that works for you.

me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough.

250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.

I wonder what exactly these laws prohibit. Like, does it apply to a fully encrypted cold-copy on Amazon Glacier? If you don't store the encryption key outside of UAE, I'd say this isn't even the same data that gets transferred to the third party, it's just some random blob. But I have no idea if the authorities of that country would agree and if it's even actually enforced for that matter, or if it's one of those laws that actually cause problems only if you follow them.

In this case, a local on-prem backup in your office-that-is-not-an-aws-datacenter would've worked and satisfy the law right?

(or an AWS Outpost thingy, assuming those things work even if the mothership is down)

For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably

a) letting customers in other areas know that their data is backed up to another continent

b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"

For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.

And witness

Backup both locally and everywhere no matter what.

50%+ of companies that lose all of their data go out of business in 6 months.

DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.

And a friendly reminder that replication isn't a tested data backup.

MENA is the on-prem capital of the world, don't worry.

... but not only in an Azure datacenter!

(The obvious option here might be an encrypted backup on some hard drives in a safe in a local office.)

It might be the you of Theseus. You really don't want to know what the future holds.

I'm from the past and I concur.

I think both of these things are (very) often true,

You think most countries have one data center or less, and most offices have less than two hundred watts of electricity supply?

I mean, in those rooms I was arguing over the best policies to prevent access and surveillance and unlawful processing, and what the potential cost-benefit analysis was. And what I was arguing against was an assumption that physically compelling all companies -- or worse, all citizens -- to keep their data within the borders of the host country, would protect you from these problems.

We'd have to explain that if the data was physically in Brazil, but hosted by a U.S. company, that would not stop that company from accessing that data remotely -- unless you specified that. We'd have to also explain that if you were intended to defend against US mass surveillance of non-US persons by the US intelligence services, intelligence services and SIGINT are univerally almost defined by their broad remit to target foreign nations on their own territory in violation of local law. And, finally, if you intended to use the prohibiting the movement of of data as a sanction against companies to punish them for violating data protection standards, as pre-GDPR law in the EU had as an ultimate last resort, and the GDPR often ends up relying on as a last resort, you would find that multinationals are more capable of putting up servers in your home territory and continuing to serve your citizens than they are of substantially changing their practices regarding data processing.

I don't want to sound nihilistic about this -- regulations can exist in these areas. But it's those politics and policies of the institutions with control over the data that are the most important part of this: not where the bits are kept. Especially when those bits are encrypted, and the keys and access controls are elsewhere.

You can also create reasonably diversified data silos within a country.

I generally agree, although if a small country only had half a dozen or so redundant data centers then it would be relatively easy for a powerful adversary to wipe out all of the data centers and potentially have a significant economic impact on that country.

Having a backup data center in an ally country might make sense. Kind of like how I keep an encrypted backup hard drive at my parents house. Whenever I go to visit I pull it out and backup my laptop there too.

  > I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws.

That's why the input of actual specialists in a field should be the one drafting the policies. I'm glad that professional lawmakers exist, I personally couldn't draw up a proper par if I had to, but they are not and can not be specialists in every field.

you know this is why china and russia were stealing data, so that they can provide you backups if you lost yours /s

I mean, it was clear and open that USA will spy on any data stored in there. Because foreigner do not get legal protections.

And second, the USA is in the middle of power grab that completely ensures any data stored there will be taken hostage wherever suitable for "negotiations".

and you don't upset whichever prince decides to compete

They're not firing nukes yet. Missiles don't blow up a whole building, let alone a city.

Check if the requirement is "only in the uae" or has to have the primary copy in the UAE

He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.

See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.

Bezos hasn't lost anything from this. He's only gotten richer.

I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?

It's not really a black swan event just an unlikely one. If it were one it would not have been something people talked about before.

Long time no see.

2013-2014 would be about right, they were still in SEA when I joined in 2013 but were actively working on decommissioning it.

It applies subnationally. Think about different gambling laws per US state, and if you’re running the infrastructure for that across state lines.

"Black Swan" would be a good name for an attack drone

they should provide and option for multi-AZ with drone defenders now

The me-south-1 (Bahrain) Region was three AZs: https://docs.aws.amazon.com/global-infrastructure/latest/reg...

"The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand," AWS said in the status update

I think a lot of people have built those services. But doing it right is more expensive, and people end up choosing the $5-$10/mo option over your $30/mo+ that does it right. Multiply those numbers by whatever multiple you want for higher end stuff.

Well, one of the things I would be very, very focused on is LLM optimization, so that is something that could be mitigated, at least. Especially since I would be building with it, certainly, I would try to engineer it so that it was to Mr. Claude / Mr. Gippity's taste

I mean I would be limiting it to a very specific use case, and the flat fee would be "plus a percentage of the underlying", something like $50/mo + 20% of the AWS spend underlying.

And for that, with a simplified use case, well, they can scale up to $20k+ a month if they like, that would be ideal, and if they need a more complicated setup or enterpriseyness, migrate off with my blessing and available-not-required hands on support (again for a reasonable fee, maybe $10k if you want the white glove).

Or encounter their CLI and it's lovecraftian combination of positional arguments, named arguments, piping, json documents, etc, distributed largely by random lot, as far as I can tell.

No. If you're the leader you should know what you're leading. It's literally the job.

But the problem is right there: "non-technical" leaders. Never work for one if you can avoid it.

And for a time, it even seems true as long as the natural disasters that impact various customer offices happen to not impact that one particular data center…

I mean, can't cloud kinda solve all of those if you just use the right things?

That's fair. The only reference was about one datacenter blowing up. You can't have unlimited redundancy.

Yeah yeah I know about 3-2-1.

But when I put data on normal S3, isn't my money supposedly paying for triple redundancy?

Look, here's the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

"S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region."

As far as I've heard before that's not with parity and three zones means three copies. But when I search now I see things about 5+4 parity, any insight here?

Get some training. You dont even understand the core concepts, and the difference between a data center and an availability zone...

Pretty much this, it’s your responsibility to use their tools to make sure your data is managed in such a way that any data destroyed is already elsewhere before the event.

which is not necessarily free

Not just in terms of service costs, but in time and complexity. In many cases building out that complexity is complicated and difficult. And sometimes the functionality you need isn't supported in the regions you use.

War?! In my Middle East? Preposterous.

This is almost always the case. It's one of the frustrating things about the software industry; because everything is much more complicated than the customer is able to comprehend, a software company can promise anything and the customer can't actually verify.

So any software company/project which actually took the time and effort to fully handle the enormous complexity, they can't sell themselves based on that fact because every other company (who didn't invest the effort) is also claiming it and the customer has no mechanism to verify the claims until some major rare event occurs.

And most of the effort is required precisely to handle those 1% of rare situations.

Perhaps, cheapness is always an factor but I'm potentially reading it as the customers perhaps not wanting data to move outside of the country and with AWS only have one datacenter in said country produced this result.

its also the fact that a lot of fundamental systems work in trade offs.

Do you want performance, or correctness.

Well, if you want performance you use write through caching and in the case of distributed storage: more nodes confirming the block before returning. Huge performance cost.

Outsourcing this just means someone else makes these tradeoffs, they will prioritise the general case- and they’re even more incentivised to move the needle towards things that are most visible to the end user.

In this case, performance.

You won’t notice that theres a third commit server off-site (unless that site is bombed), but you will notice slower writes- and the general case says that people will express comparative dissatisfaction with weaker performance and use it as a justification to use another provider.

its not true. If everyone actually did the bare minimum, everything would screech to a halt immediately. In fact people doing that imo is part of what causes the decline of empires. You can, in fact, trust people further than you can throw them.

Its a cousin of the mindset that the reason people don't steal is because they think they will be caught and rationally weigh up based on the value they gain and the chance of loss that its not a worthwhile action.

No, most of the time people steal because they think its wrong, and they dont want to do it.

Public trust is a real thing and varies massively by country. America is notably extremly low on this metric

“It’s not the cloud. It’s just someone else’s computer.” - MM

It's probably a dedicated government type thing where the data is housed seperately from normal AWS

Well, listen, you do know that even when the largest most ambitious and most sophisticated things are shipped, each owner of each specific part (could be many many owners) basically , to the best of their ability, prayed that nothing particularly bad happens when it’s shipped off. Truly, that’s the best a mortal human can do, pray their part doesn’t break.

So then that big thing comes to you. It’s all kind of … held together by a prayer …

Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.

It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life.

For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.

Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are shared across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't shard across data centers at all, only across racks within a data center.)

Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.

Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).

But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 has durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!

This is focusing in too close.

Bezos has been working with Trump. The instability has been hugely exacerbated by Trump. The issue lies at home.

They did build redundancy but most of it was bombed.

If they're region-locking data appropriately (i.e. for people to comply with domestic storage / gdpr-style requirements) they really can't. Bahrain is only about 300 square miles / 80k hectares.

They could, of course, open other regional data centers in other countries, or say "data in this geo zone may be in any of X, Y or Z" countries, but for the latter that pretty starkly limits some of the major customers they'd have, I would guess, and for the former, well, they have other geo zones already, so if people weren't replicating to them, I'm not sure why adding me-east-1 me-west-1 me-central-1 would fix that issue, they just wouldn't replicate there either.

Multiple zones impacted.

stored exclusively in

Now I'm thinking about legal/contractual rules that might force that kind of geographic risk.

I mean, logically you could have the Allowable Location send pre-encrypted backups to anywhere in the world, except (A) laws and regulations aren't always logical and (B) you still have the problem of keeping the decryption keys somewhere safe without leaving the key jurisdiction.

Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract.

Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.

aws egress fees: the real hero we didn't know we needed

> AWS promises to keep data safe without requiring cross region redundancy.

A perfect example of a claim they never made.

But AI chatbot advised him that was the case!

Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead.

Now, there are a few things to consider:

- AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data

- You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data

When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.

Clearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment.

He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions...

AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list.

And their Well Architected Framework and other docs make this plenty clear:

"It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)."

Also...

"All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region."

And also for single-Region / Multi-AZ architectures:

"Where possible, you should also copy data backups to another AWS Region as an additional layer of protection."

"AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters."

https://aws.amazon.com/blogs/architecture/disaster-recovery-...

Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross...

This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...

articleAuthorId: 30dacdbc-6a8a-11e2-9d12-0018fe8a00b0

articleAuthorName: cbsnews (hidden byline)

articleSecondaryAuthors: n/a

articleEditors: n/a

Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.

I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".

Uh.

Uh-oh.

It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.

The wording certainly seems very careful:

"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"

That's the excuse they'll say, yes, then we quote back to them "eleven nines" and they come up with an excuse for that too

Guess those multi-AZ promises have an asterisk when actual missiles are involved. Makes you double-check your own off-site backups.

"..even if something happens only once in a billion requests, that means it happens multiple times per day within S3."

but one in a trillion...

Okk

Yas

Iii

The data center layout should look something like this:

    me (Middle East)
    ├── me-south-1 (Bahrain) DOWN since 2026-04
    │   ├── mes1-az1 DOWN
    │   │   └── mes1-mct1-az1 (Oman, Muscat) ???
    │   ├── mes1-az2 DOWN since 2026-03-01
    │   └── mes1-az3 DOWN
    ├── me-central-1 (United Arab Emirates)
    │   ├── mec1-az1
    │   ├── mec1-az2 DOWN since 2026-03-01
    │   └── mec1-az3         DOWN since 2026-03-01
    └── il-central-1 (Israel, Tel Aviv)
        ├── ilc1-az1
        ├── ilc1-az2
        └── ilc1-az3

Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.

If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.

See also previous discussion: https://news.ycombinator.com/item?id=49033240

us-east-1 is finally looking pretty stable for once.

Kind of like building your house on a floodplain and claiming insurance for a flood. They'd still say it was an act of god.

https://en.wikipedia.org/wiki/Act_of_God

Seems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.

There are many regulations over there which prevent data from leaving the country.

Sometimes those rules can have serious consequences.

Since roughly zero of the articles I've seen link back to the source of their quotes, here it is: <https://status.aws.amazon.com/#multipleservices-me-south-1_1...>

Everything not saved will be lost.

We collectively gave this company trillions over the years and they still can’t get it right.

I wonder if this will cause a mini cyber insurance crisis. I don't think any of those data loss plans have been tested at scale.

Yes there’s never been division or wars in Europe and the EU will always exist…

Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy.

I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left.

And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.

They can recover most of it? I assumed that those AZs had been offline for so long because they were like gone gone.

My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical report of why they can't recover things.

So many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.

If all your backups are with AWS you don't have backups at all.

I suppose this is why Microsoft and Google want to build data centers in the Netherlands. Its not cheap but it is safe.

I wonder what kind of "some data" is: military installations in Arab states and Israel?