Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
Same - just had some malicious bots running through my platform last week and really thought they found a security hole after all. Even though the amount sounded ridicoulus, I got quite nervous and a very bad feeling when I logged-in AWS and saw that price.
Same here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same.
It's been 2 hours and I still haven't fully calmed down.
Same. Cold sweat for about 20 minutes. Even though I saw the service health notification, I still spent the last hour trying to find where my storage spiked. In any case, I'll be tearing down plenty of stale infra after this!
The should pass a law saying they should have to pay you the amount over the correct bill as compensation; I bet they'll stop making mistakes like this pretty quickly after that
Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.
This is embarrassing for Amazon, but I’d take laughably wrong over subtly wrong any day. If the bug made bills 20% higher I probably wouldn’t have queried it.
Mine is showing $241,946,798,744.75. I know it will be reverted, but for a brief minute there I suspected someone compromised my account and triggered rust rewrite of everything using thousands of agents via Bedrock :)
Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error.
You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
Yep, the truth is nobody cares when people start submitting dozens of PRs a day with a bunch of AI-generated code reviews attached to it, all saying everything looks good. I'm witnessing this happening at my workplace right now: Sr/Staff uses Claude to generate 10 pages of design document, Jr uses Claude/Cursor to generate a humongous commit based on this document and create a PR, then bunch of automated AI-based code reviews kick in and say this looks good, another Sr/Staff takes a glance and rubber stamp it, while looking at the company's stock value and/or OpenAI/Anthropic job description.
The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.
Either way it shows their QA and testing procedures are incompetent. It's just not acceptable for a utility like AWS to move fast and break shit. Should make you question whether it's safe or advisable to use any of their services.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't be charging anything. So 3015000 must be correct.
You should charge your customer 3015000 thousand dollars.
'My absurd statement doesn't sound right, so the "opposite" (assuming it's well-defined and unique) must be true' is peak LLM logic. You can tell it was trained on Reddit commentary.
I typed it. I'm afraid my biological neural network has been trained by reading too many chains-of-thought. I might have added an extra 0 by accident, I didn't double check, but that just makes it more AI-like, really.
Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.
I've found personally it's better to use AI to build a deterministic script for calculations like that. (anything that manipulates data should be a script not an AI).
Because I was already doing categorising and analysing same data with agent and I had my session open already. It should've been an easy task for an agent, right?
I hope they send out some free credits at least. I imagine quite a few people got a real fucking scare today. They haven't even sent out any corrections yet.
This is probably going to push me to completely close a couple of AWS accounts I setup when doing training courses so I could get certified (mandatory requirement from my work).
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
It was over $500k in the email I got. Not a fun experience. My hands were trembling.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
Well, even if AWS tried to charge my credit card on file for $500k, it would definitely not go through. Then they’d probably either forgive your bill or just ban you, since I imagine the threshold for taking people to court is fairly high.
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
logged in this morning to find a bill of $595 Billion... heart rate went through the roof... then I noticed the open issue, phew! nice one guys... you got me there...
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
Even though it's just a bug, being charged $595B on a platform that is known to cost spike, reminds us that we're not in control of the platform, or our company's expenses.
Whatever you do, AWS, don't post a related service health alert site wide on the console. Heck, don't even post one in the billing module. We wouldn't want to overdo the alerts, especially when we already have one being displayed to market the new FinOps Agent in Public Preview.
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more
information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
Don't worry. With so much debt banks start to treat you with respect. /S
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally.
These ridiculous amounts? Not so much.
I don't want to say this was ever or will ever be a good idea, but the reality of warfare is a lot of the time dudes were just running into an alley and firing off mortars without trying to look or think of what they were shooting at anyway. I doubt the Taliban gave a shit about false positive rates when they were cutting the hands off of anyone who voted. They got the point across either way.
US Navy now doesn't care either. Using Palantir's Maven Smart System, which incorporated Anthropic's Claude AI model, to identify and evaluate targets - which blew up the girls elementary school in Minab.
We retro-fitted a Terminator T100 model with the brain of the latest LLM models, and then gave'em 2 shotguns...and, you'll never guess what happened next!
Well, actually i guess you can guess what happens next! lol :-D
Not because of his net worth (he is probably worth 200bn if it came to liquidating) but because he will pull some trick to get the money. Maybe he will chuck SpaceX under debt, draw a dividend then set up a 1000 year payment plan with the bank.
I saw this in action on a smaller scale. In a past job, my wife organized events for a decent sized company. After an event, she'd typically have a $300k+ balance on her corporate Amex. When she went on maternity leave, the person filling in for her job neglected to actually pay the bills, so when she returned there were quite a few emails and voicemails from Amex regarding the over $500k balance.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
I think that this might reflect more on Amex to be honest.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
Interesting. And hard to square with my perception of banks as completely mercenary and ruthless. I had a decade-long personal boycott (I know, LOL) of Amex after they, because, with otherwise perfect credit, I forgot about a $30 department-store card bill and got a 30-day-late mark on my report, Amex got spooked and abruptly closed both my never-late accounts with them (which were at or close to 0 balances). This was around 2008 though, so perhaps this was a genius algorithm designed to try and detect the very first whiff of consumer defaults, so they assumed that $30 was the first domino to fall of my personal financial ruin that could lead to me charging my accounts to the max and then going bankrupt.
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
Most banks are completely mercenary and ruthless unless its in their incentives to other outcomes. Incentives lead to outcomes and mostly AMEX's incentives are in being the most trustworthy because their real targets are mostly billionaires/heavily influential people.
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
I once got a credit card statement that said estimated time to repay ....... more than 100,000 years. It was discouraging but I did pay it off. And sooner than estimated.
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
I did too, those awstrack.me URL's look super suspicious and I hadn't seen this alert trigger before so didn't know what to expect.
At first I was sure it was a phishing attempt. Then went to the console (not using those links)
Saw there was an outage where the console was wrong (no mention of email alerts)
Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong.
Looked at some logs, realized the incident text was just not exhaustive on the impact.
Went back to my cup of coffee.
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go
live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
In this role, you will own end-to-end bill run execution across all AWS partitions, drive the technical vision for autonomous billing operations, and build the team that ensures every customer receives an accurate cost estimated in minutes ...
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoicing.
I severely doubt the world ever gets to such a point that the entire world melts into AI hallucination. And token consumption depends on so many other things, it's not all that deterministic either.
Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue.
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
I don't know how cloud services count usage, but this is certainly not true for telco. I manage several fleets of hundreds/thousands of SIM cards (mostly IoT/M2M applications), and almost every provider counts the data traffic per byte. Different business and use case, I know, but still.
I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion of AWS. If a billing AI watching costs in realtime could detect, say, a lambda loop in the first 10 min and either alert the customer or kill it, that would make AWS feel a lot safer to use. Enumerating these conditions and fixing them individually is a task that Amazon has proven incapable of achieving. An AI watchdog layer might be the perfect shortcut to addressing all of these problems at once. Because it’s well-trodden territory that AWS has so many multi-thousand dollar foot guns that make it really scary to use as a hobbyist or small business on a tight budget.
I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
This is exactly the sort of thing that’s not possible, though. An AI will not be able to detect a “lambda loop” because it will look exactly like a “successful lambda rollout”. This sort of watchdog would just as likely shut down the wrong things and make AWS feel a lot less safe.
When I was at AWS, they famously required an extensive "CoE", correction of errors, or post-mortem, in an instance of over-charging a customer $0.26.
The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.
I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.
It doesn’t matter which domain it is, AI should be utilized at anomaly / patterns detection and prototyping, with goal of a prototype to become coded as discovered pathway.
I don’t understand why companies still place AI in their system at authority-forming level, if anything it should sit at least behind single guard / validator that is explicitly coded and tested, never in front of presentation layer.
If you were a business maybe you could claim for the emergency on-call time spent diagnosing, but you'd probably still lose AND amazon would fire you as a customer.
Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.
We should just go with numbers in general in order to play it safe. The new guidance from legal is absolutely no numbers on invoices for liability reasons. Similar to the removal of actual math from math class we're going to let the experts figure out how to implement it.
In order to successfully sue for emotional damages, you have to prove quantifiable damages. Usually that means if you ended up having to get therapy to deal with it, you can sue for the cost of the therapy.
To paraphrase the old joke, if you wake up with a $78,000 AWS bill, you have a problem. If you wake up with a $78 million AWS bill, Amazon has a problem.
Mine was just over $1 billion. I thought the email was a phish at first and almost trashed it, but it seemed otherwise very real, so I logged in to AWS just to check. Then I almost had a heart attack when I looked at my billing estimate. Eventually I saw a banner on the support page, but my question is why wasn’t there a banner on the page SAYING I OWE A BILLION DOLLARS?!
The finance fun noone knows about. I knew someone who did this as revenge. Sent some asshole a fake invoice for US$stupidmoney for a couple of months which said asshole ignored, then sent a certified mail 'I forgive asshole US$stupidmoney', and copied the IRS. IRS went to collect on the unpaid taxes, and if you've never been there, the IRS does not play when it comes to that sort of money. I don't know how it ended, but there were multiple lawyers involved, so it was expensive no matter what.
Like kgwgk already said - people post the occasional horror story where they run up a huge AWS bill, either due to a run-away process or getting hacked. It seems to almost always be about Compute, not Storage.
Storage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”.
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
Oh, I also don’t think I’ve ever seen that but I’m not surprised. Even if you could steal a huge amount of storage, filling it with data would take ages and the cat and mouse game of moving the data as hacks get uncovered would be untenable.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
I think they already tried rewriting billing with AI. Very smartly they only tried rewriting the estimator first. This post is about the outcome of it.
Weekend is too long bro! With AI they can weite 50 implementations in a single prompt. They clearly are using tooo many humans and that's why they're lagging
and they also imply deleting all your data so that you don't pay for storage.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
How would that work? If you only pay for one day, your access (read and write) gets cut off after the first month. Then if you don't pay the bill for the entire grace period, your data gets deleted.
Not only can you not set limits, even the alarms are not real time. So it is entirely possible to get on the hook for terrifying amounts of money and not know until it's all too late.
Scared me even though it was obviously a bug once I stopped to consider the magnitude ($bn). Very unfriendly that they don't allow for hard spend caps; closed my mostly dormant personal account as a result.
$28m actually seems worse. If I wake to a $100b bill, that’s obviously a mistake. If I wake up to a bill in the millions then my first thought would be “oh no what did I do wrong, this will ruin my life”
My estimate was only $21M (vs ~$0.01 average bill). Wish I had checked status sooner and saved myself the panic!
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
same process, same near heart attack - especially some days ago I had malicious activity on my platform and thought that now they actually found something.
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
A friend of mine went through the same - he got an alert for bill over budget, he logged in and boom 107M$.
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
117 billion us dollars. Eat that GDP of Kuwait! But yes I have never scrambled so hard to try to get on the phone with someone at AWS in my life. Terrifying 10 minutes until I found that banner on the support page. It should be front and center on the dash, not hidden away. And in yellow.
Ask for some leniency. Let your account rep know about your budget difficulties and ask if you can make good faith payments of a few billion per month until you get back on your feet.
Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
This number is obviously absurd but for other normal amount (say 2 million which can definitely happen with a mistake), can't they claim it in court that will bankrupt you even if your entire worth is just 100K for example?
I had it for three months, and each month it was unable to deliver pills before I ran out, so I cancelled and switched to the brick-and-mortar pharmacy down the street.
A year later, suddenly Amazon Pharmacy starts sending unauthorized prescription refill requests to my doctor.
I still have the account cancellation confirmation e-mail from Amazon Pharmacy, but Amazon won't close my account. Amazon's account rep says it cannot close accounts for "legal reasons." Bullshit. He can't say what the legal reasons are, or point me to a document stating these conditions.
Now my doctor's office just ignores all refill requests from Amazon Pharmacy.
I went through a two month long fight with AWS over a compromised account. I lost 20K even when my previous usage for the last 3 years was like $20/month, and I had more invoices coming worth tens of thousands of dollars. AWS refused to help until I stopped the breach myself, which required me to spend several all nighters after work learning how to script for AWS infra. They also refused to close my account after the breach was under control. They asked me to sign a shared responsibility agreement before they could look at my case, which I refused to do. I finally contacted the AG office in my state and they email AWS directly. In less than 24h I had a AWS manager calling me to fix my account and issue a refund for the 20K. Still, they refused to close my account after all this ordeal, apparently there’s no way for you to completely get out of AWS once you are in. It’s the shittiest business ever.
I had a similar situation where some dormant account was still charging my credit card.
The account was probably real, made for some purpose 15 years ago. I had ignored the charge for years because it was like $7 a month. Then it went to $300/month, making it worth the time. I could find no invoice email and none of my AWS accounts lined up with the bill.
I tried contacting support, but without the account number involved they had no way to help. I disputed the charge, the bank refunded that month and then went right back to charging the next month(my credit card helpfully accepted the charges despite the dispute).
I had to cancel the credit card entirely to make it stop.
In unrelated news I just hit my target for S3 revenue (projections). Promotion meeting locked in for tomorrow (fastest in the companies history), looking forward to being a L2 Amazon employee.
You really should get your spending under control. Unfortunately unless you become one of the real people class through a large lottery, it sounds like you owe the rest of your life to AWS until you can pay off your debts for being so careless.
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
I had Hermes managing mine, and it made a partial prepayment to help smooth out the bump in my account balance. Unfortunately Billing Support say my $17.4B refund may take up to 10 calendar days to be processed.
My personal photo backup S3 account, with a budget limit of $10, now going to cost me ....
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
I'm not a betting person but I am looking forward to the postmortem, whether or not AI was involved, and what their code to production verification stuff looks like now. This kind of thing should have been caught by automated tests.
There is an enormous amount of money invested in trying to prove the opposite, for this reason not only they will never admit it, they will actively negate AI was used. It happened before:
10 layers of approvals to get a reasonable change implemented, but this shit gets pushed to production instantly. AI is driving leaders insane - they actually believe they can stop operating with humans and take all of the money. Remember your pay is transactional. Never give them more than written in your contract, and there is no shame in playing dirty politics to advance within.
Woke up to a 100 billion dollars in S3, which is above the USD 4 alarm threshold I had set (I pay $0.55 monthly) . Some AI decided to prune the most impressionable of us.
My immediate thought was that someone managed to get an access key, stuffed a ton of data in there, and then was doing a ridiculous amount of egress. That was the only way I could conceivably coming up with anywhere near that much cost.
I doubt AWS has capability to do anything to generate $2T of charges, even if one user maxed every billable capability of every resource they have.
If I got an email with that amount I would've just laughed at the incompetence that lead to a bug that lead to me getting the email. But I write as a backseat email recipient.
They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.
Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account.
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
Airlines are all subject to a lot of the same factors, but there are unequivocally better and worse performers in terms of on-time arrivals, by a lot. Take a look at the Air Travel Consumer report for details.
unequivocally better and worse performers in terms of on-time arrivals, by a lot.
I don't think it's a very good metric to compare them with, though. Those numbers are going to be heavily weighted by conditions at their hub and focus city airports, and speaking more broadly, the cities they serve. Meaning, things that aren't really in their control, and which are either in your control (often, you get a choice of which connecting hub to fly through), or is a fixed risk to you because of your origin/destination, no matter what airline (e.g. if you're flying from/to Orlando in the summer, expect a risk of thunderstorms interfering. Or snow in winter in Chicago.)
If I'm travelling from, say, LAX-PDX, United's systemwide on-time performance is not the data I'd care about, because it's probably affected far more by the conditions in their O'Hare and Newark hubs, which I'm not visiting.
I've had a mysterious Neptune cluster appear on my billing. Never used it, no API key access (or IAM instance profile access, OIDC etc), nothing in my console shows I've ever had one in any region. Raised a case with support, they ignored it.
Will wait for the RCA, the update says that they will resort to last known estimate as of 15 July. I’m guessing that would imply that the bug is at a lower level, write or an ingestion path.
This generation is too entitled! He should some learn responsibility by paying the full amount; otherwise Amazon should delete his services/data. Consequences!
Ditto on the heart attack. My cost estimate for the month is currently $223,509,270,216.17. My girlfriend suggested contacting Elon for help. Glad I found this thread. Maybe I should create new keys anyway, this stuff freaks me out.
You don't need hours to recalculate billing. You need to go back to basic algebra.
Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.
I almost had a heart attack because of this. I was like, did I mess up my API management? Why didn't I just use Lightsail? Those were the thoughts running through my head.
My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.
My bank account barely has enough for next month's rent.
At some point my role was to reduce our startup’s AWS bill. I managed to keep 7 figures on our books instead of handing it to AWS. But a message like that would have given me a heart attack in those days.
Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.
Thank you so much! I just woke up, and saw budget alert email for a dormant account to use $434,896.90. I haven't gotten so awake so fast in such a long time.
If I can expect to be penalized for not paying my legitimate bills, companies should also be penalized for failing to implement common-sense reasonable safeguards that prevent them from slapping their consumers with such absurdities.
Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
When someone says "I nearly had a heart attack," it's _highly_ unlikely that they actually nearly had a heart attack. I don't think the chances are good that anyone actually had a heart attack.
While I agree that this phrase is most commonly spoken in a figurative sense, folks with marginal heart health have a much greater than 0% chance of having something of this nature trigger an actual heart attack.
Three billion dollars sounds about right for a free week of Kiro in the default Agentic mode. We usually see slightly higher numbers, so I wouldn’t be too concerned.
It's a good job it was off by such a large amount, or I might have panicked instead of writing it off as a phishing attempt. I had an email saying my $7.50 budget had been exceeded with an actual cost of $3bn.
My Budget is 10$.
The month forecasted cost associated with this budget is $182,278,249,263.06.
Even though I new they could not collect the whole amount, I wondered whether I was hacked.
I closed the account, it was an old testing account anyways.
I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching servers to mine crypto which AWS somehow was unable to flag and block, and let run for months.
Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.
None of this can happen unless you first cede control.
They are also not all that limited. I used a Privacy.com card for a New York Times trial subscription, which of course I could not cancel online, so I just let it expire. A few days past the renewal date, I saw a $waytoomuch charge on my card.
I wrote to Privacy, and this was their reply:
Jun 1, 2022, 5:21 PM EDT
I've been reviewing your dispute and wanted to touch base with you to explain what happened.
It appears that the disputed charge is a "force post" by the merchant. This happens when a merchant cannot collect funds for a transaction after repeated attempts and completes the transaction without an authorization — it's literally an unauthorized transaction that's against payment card network rules. It's a pretty sneaky move used by some merchants, and unfortunately, it's not something Privacy can block.
We are refunding the transaction to the original funding source used for this purchase. You should see this refund reflected in your bank statement in the next 5 business days. However, please note that since this merchant uses force posts, they will continue to do this monthly until the subscription is cancelled. Please contact them to have the subscription cancelled.
Let me know if you have any questions or concerns!
And utilities are typically natural monopolies. They're good candidates for regulation because they're essentials and they don't have competitive forces to keep them behaving reasonably.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
I wouldn't expect their detection of hacked accounts to be 100% correct. Sure, it might be obvious when a human takes a look, but humans can't proactively look at every account's usage.
For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
I got a 20K bill once and it was actually drafted from my bank account.
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
This is generational debt for a Kardashev Type III civilization. If there are individuals who are being charged "the whole of the GDP the human race has ever produced", then Amazon is claiming the rest of the world owes them a fairly significant portion of the galaxy if you consider the entire billing claims being made here.
There have been some big billing errors in the past but I wouldn't be surprised this is a new record. I know I've seen people charged around 92 quadrillion, which is a rounding error from underflowing a signed 64-bit integer with pennies, but those stories are usually isolated individuals. It sounds like the aggregate error here may comfortably exceed that. Hard to tell how widespread this is; as a data point my ~$2.00/month spend on S3 seems to still be billing correctly, so it must not be affecting everyone. But a few trillion here and a few trillion there and you get up to those quadrillions pretty quick.
Well, they publish unit prices for everything, so you could just get to counting. Whenever I've had to do cost estimates, you estimate how much AWS resources you need and then times that by the unit price.
AI billing audit startup Vaudit reviewed $34 million in AI invoices submitted by 60 enterprise customers and found approximately $1.7 million in mistaken overcharges — a billing error rate of roughly five percent.
As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
Well, for my case, I was paying $0 (Exactly, I managed to hunt down and delete every last resource in my account a few months ago). It was displaying $430 million for me. I don't think that is 0*2^30.
Huh, that is odd. Working backwards, that would be ~ $0.40 originally. Wonder if that’s also flat out wrong or if they’re doing some kind of currency handling that breaks when you start dealing with huge multipliers.
Almost all of them have one number that appears four times, and one or two that appear three times
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production.
Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.
Tale as old as time. When I was coming up it took a $20-40/m investment to get a "dedicated" server that you could start tinkering around on. When you couldn't afford that, you bricked the family PC trying to figure out how to configure your own LAMP stack.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
No not really, nobody has to be using AWS itself per se to be honest, there are still alternatives around.
I mean I own a 7$/yr vps that I pay with crypto and my stack usually involves golang/rust and oh I also self host my mail for temp-mail esque things on that server and I have some other servers floating around as well.
Its just that everyone uses AWS which is the wrong issue here to be honest. If you ask me, aside from the ram pricing caused by AI inflation, things are pretty good. The issue is that not many people know about server providers and other things and default to AWS/Azure etc. because big companies and small companies alike are using them.
Here's a global (interactive) visualization of around ~250 providers, 224 locations and 800 distinct links in 60 countries that I have made if this space interests ya: https://buyvds.net
In moments like these I'm reminded of all the people who have committed suicide due to billing errors. This is completely unacceptable. These sorts of errors must _never_ happen.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
Any individual human (or frog, obviously) is getting out of the pot when it gets uncomfortable.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
some things never change. Pre AI I was always shocked that such large and complex systems actually run as well as they do. Especially after getting to see how the sausage is made/works.
It was the mid 2010s when I sensed a lot of SaaS becoming popular. Just host your ticketing systems, your IT management planes, your security management consoles, your SOC, all off-premises.
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
I'm sure some businesses are considering moving back to on-prem, but for many, I suspect the cost to find onboard, and pay the SMEs to keep those systems running well enough to not fail due to one reason or another isn't as appetizing to them as the ability to offload that work, along with the legal responsibility.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
One thing that you need to understand is that the usual business manager absolutely hates depending on technical expertise, and that the modern corporate world is fanatically anti-intellectual.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
Also, the 130 IQ Engineer usually can't afford to wine and dine the business manager the way Amazon can. Or dangle the possibility of a cushy yet prestigious sounding job in front of them.
Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who have literally stopped communicating with any of their own language - meaning every interaction they have with anyone not in person is made by an LLM. It's a massive threat to our brand and has got to stop. I can't imagine what companies with thousands or tens of thousands of employees who have really been riding the LLM train are going to have to deal with. This thing is more of a virus that exploits human laziness than actual useful tech.
I realized recently that Whole Foods no longer automatically and reliably detects your Chase Amazon Prime credit card when paying. So they don’t give you the discounted pricing automatically. I wonder how many customers are checking out the way they always do and are paying full price when, for years and decades, this worked fine.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
Relevant to this, I've recently noticed a trend of mass tort cases being opened up in the past couple years, and they seem to do very well. The way these seem to work is attorneys identify a company who has clearly ripped people off, and what I presume is a repeatable way to guarantee a win (thus translating to a guaranteed settlement offer). Then they advertise for eligible clients, sign those clients individually to contingency agreements, and run the playbook. A couple months ago after signing up for one of these, I received a check for about $350 (after the agreed-upon 40% attorney fee), from Ticketmaster, and I had another one related to AT&T. It took about 10 minutes more effort from me than a typical class-action settlement, because I had to e-sign those representation papers.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
There are a hundred small things like this that seem to be popping up in what used to be simple and reliable systems and as much as I know they aren’t ALL because of vibe coding I can’t help but wonder how much is.
Weirder is what happened a day later. I got an email that said my Chase Amazon Prime credit card was being re-associated with my Amazon.com account.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural
area at the moment and need delivery.
Me too, hello all. I've got 59 Million dollars in billing threshold reached email from AWS. And felt the same as others, and after half an hour of investigation I saw their message on top of the support page. I have 6 mb for static website stores in S3. No other resources.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
A couple of my coworkers think I’m nuts for watching cost explorer so closely but
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Huh? Apparently Amazon has never paid dividends, has done much less than 1% buybacks, even as CEO Bezos was getting less than an engineer in total comp (but did get loans or something).
Apparently Bezos owns about 9% of common stock - no dual class shite.
So your yacht spending sounds like a hallucination: if I'm missing something, please correct me.
Amazon's weight: about 3.7–3.8% of the S&P 500. So he isn't even that overweighted (especially as the market doesn't like owners selling down - bad optics).
Bezos has spent about 28 billion for Blue Origin by selling Amazon stock - not using shell games. He has comparitively spent a few percent of that amount on superyachts.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.
There's a certain obligation on both sides of a contract to pay attention.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
No argument, but fuck-ups happen, and get fixed more quickly and easily when people are paying attention.
I once got a monthly water bill for ~$35,000 at a residential, single-family home. Good thing I was paying attention and looked at the bill before the auto-pay bank draft hit.
Banks and financial institutions are the same. If they haven’t issued you a monthly/quarterly statement yet, they can just apologize and tell you the numbers are wrong please wait for the statement. But it is a major issue if an actual statement has the wrong numbers.
If AWS was a predatory mobile gacha game, we'd get 300 apology gems as credit to our accounts for this mixup, to help us in our rolls for the next 3-letter acronym they release.
Clearly they weren't tokenmaxxing hard enough or weren't using the latest models /s.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress).
And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
Given the wild but apparently "consistent" numbers I wonder if we could reverse-engineer the wrong algorithm with enough data points? Maybe the proper cost estimate has some relationship to the reported cost estimate.
I wonder what's going on; they still don't have a potential solution after 7 hours and they have multiple teams on it. Never seen anything quite like this
Maybe it’s one of those absurd situations where canceling a service doesn’t actually stop the charges. Instead, they quietly begin billing you for some random add-on that was bundled with the original service. You never knew it existed, never knew it had to be canceled separately, and now you’re paying full price for a completely pointless ghost service because the only thing it was tied to has already been canceled.
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
This is just the cloud area, what if Amazon starts vibe charging regular customers because of some bug? Accounts that are directly linked with regular people's payment methods?
Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calculation should always be:
1. Current month's usage * applicable rates; +
2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
Thats not really how estimates work. The actual metering data is ingested in near real time. The metering * pricing plan is processed within a few hours; thats what youre seeing for “estimated spend” IIRC. The actual billing accumulation is done later, at the end of the cycle, because pricing has cross service discounts, price tranches, credits tied to total spend, etc.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
I generally think AWS is better than GCP and azure, but them not allowing spending caps is a big worry source for me and something that has made me pause and rethink using them. A bad click or a bad actor can create tens of thousands of dollars of spend nearly instantly and they can, and will, bill you for it. I can understand that stopping services is hard but some system would be good. For instance, if they had a two tier system where you could stop new services and active things like EC2 would shut down (but not delete) if spend is > x, that kind of thing. Some sort of 'stop the bleeding' concept would give me a lot of piece of mind using them.
"""
If you own the bank $1000, thats your problem.
If you owe the bank $1.7B, thats the banks problem.
"""
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.
In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.
Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.
There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.
If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.
That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.
From the perspective of people you manage, it's a very different picture.
We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.
So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.
The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.
I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.
Absolutely not my experience at AWS.
All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.
Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.
Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.
Some orgs also use them to extract concessions from dependencies further up the chain. Upstream service won't fix an issue that's been causing problems for us? "unintentionally" let it become a SEV, so that we can send the CoE up the chain and get Jassy to drop the hammer on that team...
Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again
And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?
The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
there's definitely this sort of monitoring in place. Surely somebody was paged.
Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
Yeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done.
You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.
I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y.
[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.
I think they're saying that their understanding of the hypothetical error is that somebody more or less configured a price as 5¢/byte instead of 5¢/GB. And if that's the case, that's not really a thing testing will catch. 5¢/byte is as "valid" a price as 5¢/GB.
There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.
Yeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.
Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
So how is it a "foot gun" to add XTS as an additional currency, for internal use only?
I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.
I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".
presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate
When talking about e2e tests, it matters where the e's are. If they are at the public interface of "the service", then, indeed, you'd test against "credits".
But ideally, there'd be e2e tests that test the consumers' experience, where the 'e's are the web interface, emails, CLI and other things consumers click on, read, enter etc. This is where a "test currency" makes sense.
Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.
Nah, this is not uncommon in AWS. I came across a similar bug 10 years ago when playing with Kinesis and nearly had a heart attack at a $2m billing alert! Turns out it was just 5c
I love how AI means that all human-written code is now perfect, and any problems are onbviously a symptom of vibe coding. Another few years of hearing this and I may begin to believe I only wrote perfect code, when my current memory suggests otherwise.
I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there.
They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
Earlier this week my slopservant implemented several comprehensive changes to a codebase.
It also wrote extensive tests to verify the correctness of the changes.
A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.
The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.
I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.
I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase.
The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little.
The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!
write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes.
If you write the tests the agents have an easier time writing the code!
The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.
Then it can write tests based on the interface, and that is usually a lot more solid.
Yeah for sure it's a bandaid fix, but it does produce tests that fail if the interface doesn't match the implementation, more often than not doing this does (in my experience).
With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.
if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.
In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.
The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.
we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D
...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode.
There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.
This seems to happen quite a bit with software companies.
I have bad news if you think banks are the pinnacle of cyber security practices...
Maybe some rare few are, but by and large they generally stink.
They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains.
So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average.
This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.
I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures.
That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
All of my science classes in the rural US from middle school onward were taught entirely in metric, and it had been that way for decades prior. Obviously I'm an adult now, and metric still isn't universal.
Until someone comes along and changes it without updating the name, or the value comes from a remote service and they change without you changing your name… etc.
Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.
Units and datetime will always be the bane of any professions ...
But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?
Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars.
I'm stressed thinking about it, like from a theoretical standpoint.
actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part
I wonder if AWS billing still uses CSV files for passing data around.
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far.
Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.
I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.
Because CSV is the "interface" that will bring your system down if not properly designed.
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.
Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.
Sure, and I would expect the COE called out the operational aspects. No one should have been in a position to be able to trigger that bug trivially (I don't remember if it came from a service, or if they injected the metering record themselves somehow, or quite what. Way too long ago for that)
Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!
Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.
100% agree with this, seems weird to call an engineer “smart” for triggering an API that allows any input but can’t handle it. I also kind of doubt it was an engineer testing vs some much more subtle issue as billing is in line for everything.
If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.
Just a matter of time if no one is escaping commas in billing files.
It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.
Give that engineer a raise and an official title of "chaos monkey".
On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.
Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.
Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.
A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.
---
¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.
I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.
The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.
The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.
I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.
Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.
Services emit metering values that arent directly tied to prices.
Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.
I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
Just got a call from the IMF president begging me to not default my debt with Amazon and offering me credit line and a plan to re-structure my debt so I don't create a global financial crisis with my default.
Yes have gotten that before the hundred billion dollar billing alert. Are you ignoring it? Unit error doesn’t do this does it? Maybe they were hir with malware?
Has anyone received any proactive communication about this? I didn't see the email until about 9 hours after it was sent out, yet I still needed to seek out information as to a) whether this was real or phishing and b) whether the amount was correct.
Seems totally irresponsible not to send an immediate follow up email to make customers aware.
You requested that we alert you when the actual cost associated with your Monthly AWS budget exceeds $2.00 for the current month. The month actual cost associated with this budget is $646,677,805.51.
Current usage: $1.70.
I called it a rounding error and figured it would wash out within a couple of days...
You know, at one time Amazon was grouped in with all of the other big tech companies as a member of FAANG, as if the company has a culture of solid engineering at its core like Google or Apple.
Amazon has made some impressive software, but there's also a ton of junk getting shipped. Customers and "partners" have to deal with it every single day. Sometimes it's "oops"-level stuff like this, at other times it's AWS regions going dark for hours.
Browse the Amazon seller forums (https://sellercentral.amazon.com/seller-forums/discussions?s...). Every day there are stories about back-end things breaking, or AIs overriding attributes or flagging random items as pesticides. The receiving errors at Amazon warehouses are incredible, and take months and multiple support tickets to fix.
I saw another post on reddit with something like a hundred trillion dollar bill. I wish I had saved that link to share it, the comments were quite comical!
$1.7 billion? That's all? I usually spend around 75 cents a month (migrated off AWS due to it being way too expensive). They emailed me with an estimated bill of $264 billion.
After 10 years of building on AWS infra I still had no idea what anything would cost. At some point it just became clear to me that they intentionally designed it this way and that complexity is the feature. I switched to digital ocean last year and really haven't had to think about infrastructure again.
Just got an alert that we would owe $12 Billion. BILLION. After getting me out of bed and a breathless 15 minutes of investigation, I found: https://health.aws.amazon.com/health/status#billingconsole_1... which notes this is an ongoing issue on Amazon's Global Billing Console.
I once got a wildly inaccurate billing estimate shortly after I created my AWS account. I could not find a out which resources that would cost me so dearly, and saw no other option to delete the entire account.
A few hours later, I got a mail saying the estimate was wrong.
Now, I cannot create an AWS account using my email address because you cannot create a new account using the same email address…
My alert thresholds are $8 and $10 because I only use aws for some backups to S3. It was reporting my actual spend as $425 billion, and forecast at $777 billion.
I thought it was a bad phishing email first, when I saw it in the console I just laughed. I killed my access keys anyway before I noticed the incident on their status page.
Checked this morning and it's all back to normal.
I'm not sure I've seen anyone with a higher actual / forecast from this. Can anyway top mine?
From a CFO perspective, this is catastrophic. With monthly billing automatically feeding monthly close pipelines, this would result in results that either need to be adjusted (without proper support) or tank performance until it's fixed on the vendor side. I wouldn't want to be in that situation.
I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
A couple of relevant links: - AWS Status Page: https://health.aws.amazon.com/health/status - Reddit Thread: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
Just got a budget alert that I owe $286,486,223.88 on a hobby aws account, almost got a heart attack.
Yea, same here. $420M+ bill, when we have <10$ per month usually.
I got 109 billion - am I the winner?
Depends. Did you also get a free heart attack?
This is real risk. Someone could really have a serious health problem.
Sorry mate, $241,946,798,744.75 for Glacier here.
I was actually in the toilet when I got an email I owe them $36,869,876,146.51. I literally just shit myself.
Mine was about the same and evoked a similar response.
I got one for 8 billion while I was eating lunch. Thankfully I managed to not vomit.
Ok, back to $0.17 :D
Same here. Usually $0.15 per month, current bill is $15.4 billion.
I went from 0.03€ to $8B.
Not only did your cost spike, it changed currency and went from postfix to prefix!
I understand people complaining about large bills, but this is over the top!
Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
Same - just had some malicious bots running through my platform last week and really thought they found a security hole after all. Even though the amount sounded ridicoulus, I got quite nervous and a very bad feeling when I logged-in AWS and saw that price.
Same here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same.
It's been 2 hours and I still haven't fully calmed down.
I did the same! I was trembling all day!
Same. Cold sweat for about 20 minutes. Even though I saw the service health notification, I still spent the last hour trying to find where my storage spiked. In any case, I'll be tearing down plenty of stale infra after this!
The should pass a law saying they should have to pay you the amount over the correct bill as compensation; I bet they'll stop making mistakes like this pretty quickly after that
Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.
I don’t know how something like this makes it to prod. That’s multiple levels of failure.
This is embarrassing for Amazon, but I’d take laughably wrong over subtly wrong any day. If the bug made bills 20% higher I probably wouldn’t have queried it.
Mine is showing $241,946,798,744.75. I know it will be reverted, but for a brief minute there I suspected someone compromised my account and triggered rust rewrite of everything using thousands of agents via Bedrock :)
Phew.
Maybe this is a new strategy to scare people into finally locking down their old, unused AWS accounts. It sure worked for me!
I got estimated costs of $56.something billions. Usually ~$100/month. My heart rate currently still sits at around 160 bpm. Motherfuckers.
I expect such incidents like this to continue. So please keep vibe coding.
Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error.
Vibecoded the billing system, raised revenue 9000%. Great for that promo package.
Oh great so 2*30=60 he only owes 28.3$ million... hehe
I guess you wanted to say 2^30 which makes 1.5$
My hunch is the HN formatter swallowed the double asterisk typical of python exponents.
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
But 2^30 is 28.
AI slop. Or just a distracted dev
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
I'd love to see the spike in their projected earnings internal dashboard :)
What if there's only half a dev and a swarm of agents after the layoffs?
That's one way to cut staff.
Yep, the truth is nobody cares when people start submitting dozens of PRs a day with a bunch of AI-generated code reviews attached to it, all saying everything looks good. I'm witnessing this happening at my workplace right now: Sr/Staff uses Claude to generate 10 pages of design document, Jr uses Claude/Cursor to generate a humongous commit based on this document and create a PR, then bunch of automated AI-based code reviews kick in and say this looks good, another Sr/Staff takes a glance and rubber stamp it, while looking at the company's stock value and/or OpenAI/Anthropic job description.
It's a shit show.
The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.
Once you lose ownership you lose interest. AI has supercharged that. It a huge matter of scale now.
Your assuming they have testers and a pipeline of regression tests anymore...
Either way it shows their QA and testing procedures are incompetent. It's just not acceptable for a utility like AWS to move fast and break shit. Should make you question whether it's safe or advisable to use any of their services.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't be charging anything. So 3015000 must be correct.
You should charge your customer 3015000 thousand dollars.
Would be funny if it wasn't so close to true
'My absurd statement doesn't sound right, so the "opposite" (assuming it's well-defined and unique) must be true' is peak LLM logic. You can tell it was trained on Reddit commentary.
Was it proof by contrapositive?
So uh did you type that out or generate it somewhere?
Number felt high so I wanted to double check and I only get 301500.
I typed it. I'm afraid my biological neural network has been trained by reading too many chains-of-thought. I might have added an extra 0 by accident, I didn't double check, but that just makes it more AI-like, really.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.
I've found personally it's better to use AI to build a deterministic script for calculations like that. (anything that manipulates data should be a script not an AI).
It was just one off task and I already had agent doing categorising with the same data so I just asked it. Otherwise I agree.
It doesn’t take any longer to have the llm script this kind of thing, one off or not.
That is literally what Excel is for. Why didn't you use that first of all?
When all you have is a hammer, but the hammer looks more like a swiss knife
Because I was already doing categorising and analysing same data with agent and I had my session open already. It should've been an easy task for an agent, right?
This error could be fixed with better typing. If you compute on GiB in a billing system, make sure it can only ever be mutated with a GiB type!
https://health.aws.amazon.com/health/status
Looks like this is a bug w/ S3
$627,487,837,871.49
I might be a winner.
I feel much better after seeing the $B estimates here; I only have an estimate of $34M so far
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
Peanuts
It's ok, I owe them 1.22 trillion.
Maybe you went over 9,223,372,036,854,775,807 twice and came back to positive.
Uhh class action incoming? $34,909,930,575.09 over here.
I hope they send out some free credits at least. I imagine quite a few people got a real fucking scare today. They haven't even sent out any corrections yet.
This is probably going to push me to completely close a couple of AWS accounts I setup when doing training courses so I could get certified (mandatory requirement from my work).
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
Excellent point!
I remember seeing a story shared by AMS RDS on linked-in that basically said... "Now with predictable pricing"
Huh?
What would your damages be? They’re not actually going to charge your credit card for 34 billion.
I mean, emotional damages are a thing right?
Not really in the way the media would have you believe.
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
I wonder how many people died of heart attack when they saw this.
I could see someone sadly taking their own life over this.
Aws has created more unicorns than any accelerators.
It was over $500k in the email I got. Not a fun experience. My hands were trembling.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
Well, even if AWS tried to charge my credit card on file for $500k, it would definitely not go through. Then they’d probably either forgive your bill or just ban you, since I imagine the threshold for taking people to court is fairly high.
In a twisted kind of way, this is an advantage of going to court in the US being SO expensive, even $500k may not be enough to justify it!
Maybe they're using too many humans and not enough AI in their software development. That must be it.
The code base is not gigantic enough they need AI to generate massively more lines of code.
But they're going to try anyway.
My guess is the GP swallowed a comma.
Your right! My misteak!
Well AWS never had bugs before.
They need the customers to pay more so they can fix the bugs. It's self-correcting.
yeah.. i just to a daily cost alert.. it was only 23 trillion dollars this month. i thought, hmm seems kind of high this month.
https://health.aws.amazon.com/health/status
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
Update as of 7:53am PDT:
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
Wait what if someones actually getting usage spiked
Hackers rejoice!
Yes, I've got an estimated bill of $4bn. Probably related to the ongoing "Inaccurate Estimated Billing Data" incident?
https://health.aws.amazon.com/health/status
Yes, an incident is ongoing https://health.aws.amazon.com/health/status
AWS has become the uber employer: before AWS, you just had regular employers steeling employee wages bit by bit by forcing work, skipping breaks, etc.
All hail the new generations of our uberployers.
logged in this morning to find a bill of $595 Billion... heart rate went through the roof... then I noticed the open issue, phew! nice one guys... you got me there...
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
Even though it's just a bug, being charged $595B on a platform that is known to cost spike, reminds us that we're not in control of the platform, or our company's expenses.
Even if it was 595 billion, that sounds like their problem.
"If you owe your bank manager a thousand pounds, you are at his mercy. If you owe him a million pounds, he is at yours."
Help, what is this number - US$87,967,679,887,258.36
That's 87 trillion, 967 billion, 679 million, and so on.
Yes, there is a known issue with cost estimation.
https://health.aws.amazon.com/health/status
Thank you. Have seen this after I posted.
Thanks, I panicked logged in and could not find the root cause of the bill.
Thanks AWS, no caffeine needed this morning!
Bit of an understatement: “The displayed billing estimates do not reflect actual usage and charges.”
"There are no customer actions required at this time."
Of course, you provided heart stress failure tests for free.
Whatever you do, AWS, don't post a related service health alert site wide on the console. Heck, don't even post one in the billing module. We wouldn't want to overdo the alerts, especially when we already have one being displayed to market the new FinOps Agent in Public Preview.
I have just received a similar alert for $ 5b
AWS on their support data is reporting this:
Inaccurate Estimated Billing Data
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
Don't worry. With so much debt banks start to treat you with respect. /S
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.
I got freaked out by the mere fact that I got a billing alert, since getting one would require my monthly spend to have suddenly exploded.
Well, no coffee needed this morning.
$103,515,940,301.79
Vibe coding billing systems is a top-notch idea :)
Hey what do you think about vibe coding weapon systems? Do you want to be my cofounder?
Sure! What could possibly go wrong?
Drones are already vibe targeting in Ukraine/Russia.
I don't want to say this was ever or will ever be a good idea, but the reality of warfare is a lot of the time dudes were just running into an alley and firing off mortars without trying to look or think of what they were shooting at anyway. I doubt the Taliban gave a shit about false positive rates when they were cutting the hands off of anyone who voted. They got the point across either way.
US Navy now doesn't care either. Using Palantir's Maven Smart System, which incorporated Anthropic's Claude AI model, to identify and evaluate targets - which blew up the girls elementary school in Minab.
Use AI => No War Crimes!
We retro-fitted a Terminator T100 model with the brain of the latest LLM models, and then gave'em 2 shotguns...and, you'll never guess what happened next!
Well, actually i guess you can guess what happens next! lol :-D
How about vibe coded health care systems.
If you owe the bank $100, that's your problem. If you owe the bank $1.7 billion, that's the bank's problem.
Not if you’re Elon Musk
Elon Musk is everyone's problem
Not because of his net worth (he is probably worth 200bn if it came to liquidating) but because he will pull some trick to get the money. Maybe he will chuck SpaceX under debt, draw a dividend then set up a 1000 year payment plan with the bank.
This joke only works if you actually impose a cost on AWS of 1.7 billion. If they just serve you a bill for no reason, it's still your problem.
I saw this in action on a smaller scale. In a past job, my wife organized events for a decent sized company. After an event, she'd typically have a $300k+ balance on her corporate Amex. When she went on maternity leave, the person filling in for her job neglected to actually pay the bills, so when she returned there were quite a few emails and voicemails from Amex regarding the over $500k balance.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
I think that this might reflect more on Amex to be honest.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
Interesting. And hard to square with my perception of banks as completely mercenary and ruthless. I had a decade-long personal boycott (I know, LOL) of Amex after they, because, with otherwise perfect credit, I forgot about a $30 department-store card bill and got a 30-day-late mark on my report, Amex got spooked and abruptly closed both my never-late accounts with them (which were at or close to 0 balances). This was around 2008 though, so perhaps this was a genius algorithm designed to try and detect the very first whiff of consumer defaults, so they assumed that $30 was the first domino to fall of my personal financial ruin that could lead to me charging my accounts to the max and then going bankrupt.
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
Most banks are completely mercenary and ruthless unless its in their incentives to other outcomes. Incentives lead to outcomes and mostly AMEX's incentives are in being the most trustworthy because their real targets are mostly billionaires/heavily influential people.
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
haha :-)
Next question we'll find out is what if you owe the bank $1.7 trillion?
That's the government's problem
The Federal Reserve? You ressurect Alan Greenspan, and tell him to brrrrr.
Cynically I wonder if this has an outcome as an unintentional (or intentional) anchoring exercise for future cost increases
I hope they're not planning for that large of a cost increase.
Wow, those price increases due to the RAM and storage shortages AI caused are brutal.
Most likely they also forgot to include "make no mistakes" instructions to their in-house LLM that deploys to production.
Rookie mistake
Same, i am now a slave to Jeff Bezos to the end of my life.
Cloud pricing has gotten ridiculous.
Host your own people. Host your own.
The old hypsters have to subsidize the new hypsters.
this counts towards ARR right? would be stupid not to
Hope they’re using 64 bits to store these prices
float will have to do it.
Pff rookie numbers, mine was 375 billion.
Small potato's sir, my bill > GDP of Switzerland. A cool $1.2T
I was at $5 trillion, on the way to $9 trillion.
I once got a credit card statement that said estimated time to repay ....... more than 100,000 years. It was discouraging but I did pay it off. And sooner than estimated.
Were you still alive after paying it off?
No, but they have the internet in the afterlife, apparently.
They do, but the latency is terrible
100K years. Now that's load-bearing ...
That's good for the credit card company, they can project stable revenue 100k years into the future.
Generational credit card debt.
“Bank error in your favor, collect $100”
Amazon, the first quadrillion-dollar company.
Golden era of software productivity they say
Look how much money AI is making.
We finally found the ROI!
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
Time to get a second job buddy.
I did too, those awstrack.me URL's look super suspicious and I hadn't seen this alert trigger before so didn't know what to expect.
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
Wanna bet the description of this job post will be updated by the end of the day?
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
It gets worst:
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
The best bit of that is:
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
If you can make the software cover the toil you save the staff for the tough cases.
They need to fire whoever is running AP and AP software development. Vibe invoicing is ridiculous for anyone to do, let alone Amazon.
And if that weren't enough, the "manual war rooms" alone should be setting off alarm bells.
This is a gold-mine. They need to get sued heavily for this incompetence.
Seriously! If I were making a joke I would say something like
But that’s word for word a 250k+ TC job in the big ‘26.
Wow:
This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoicing.
The irony is, the only purely deterministic thing, will be token consumption...
I severely doubt the world ever gets to such a point that the entire world melts into AI hallucination. And token consumption depends on so many other things, it's not all that deterministic either.
(token usage) is trending towards predictability for a lot of reasons. it's not deterministic but it's getting easier to reason about usage.
How can a random generator be deterministic?
Truth is, you can never know: https://imgur.com/random-number-generator-bwFWMqQ
Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue.
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
You have to do it when the customer list is too big to keep a counter per customer.
No you don't
A probabilistic counter per customer is also a counter per customer. Still, probabilistic billing is an amusing thought though.
I don't know how cloud services count usage, but this is certainly not true for telco. I manage several fleets of hundreds/thousands of SIM cards (mostly IoT/M2M applications), and almost every provider counts the data traffic per byte. Different business and use case, I know, but still.
I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion of AWS. If a billing AI watching costs in realtime could detect, say, a lambda loop in the first 10 min and either alert the customer or kill it, that would make AWS feel a lot safer to use. Enumerating these conditions and fixing them individually is a task that Amazon has proven incapable of achieving. An AI watchdog layer might be the perfect shortcut to addressing all of these problems at once. Because it’s well-trodden territory that AWS has so many multi-thousand dollar foot guns that make it really scary to use as a hobbyist or small business on a tight budget.
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
This is exactly the sort of thing that’s not possible, though. An AI will not be able to detect a “lambda loop” because it will look exactly like a “successful lambda rollout”. This sort of watchdog would just as likely shut down the wrong things and make AWS feel a lot less safe.
This is like half of all job listings I've read recently. And it's a decent amount of fintech that's like this.
When I was at AWS, they famously required an extensive "CoE", correction of errors, or post-mortem, in an instance of over-charging a customer $0.26.
The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.
I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.
It doesn’t matter which domain it is, AI should be utilized at anomaly / patterns detection and prototyping, with goal of a prototype to become coded as discovered pathway.
I don’t understand why companies still place AI in their system at authority-forming level, if anything it should sit at least behind single guard / validator that is explicitly coded and tested, never in front of presentation layer.
Fuck it, im in.
Just wait until the same system runs payroll and you're getting paid $1.94400 annually.
I will just tell the HR bot that I am meant to be paid 1.944 billion.
And if the HR bot is anything like Facebooks customer support - itll just let you do it!
10/10, no notes.
Or alternatively $1.9 billion
That's base, don't forget bonus and stock.
IRE which stands for Invoice Reliability Engineer.
Enterprise account . We got - 3trillion and change
-$3 trillion! That's the highest earning investment that has ever existed!
Quick do your IPO before the books update
It's true, if you're spending that much money you must be worth a ton! Just look at SpaceX!
3.7 billion. Offered to pay it in monthly installments. Haven’t hears back
same. over 2t in one day.
Yes, I am taking legal action, no doubt.
…for emotional damage?
If you were a business maybe you could claim for the emergency on-call time spent diagnosing, but you'd probably still lose AND amazon would fire you as a customer.
"fire you as a customer"
I love this framing
Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.
Big numbers can lead to stress which can lead to all kinds of disorders.
small numbers too....
:)
We should just go with numbers in general in order to play it safe. The new guidance from legal is absolutely no numbers on invoices for liability reasons. Similar to the removal of actual math from math class we're going to let the experts figure out how to implement it.
In order to successfully sue for emotional damages, you have to prove quantifiable damages. Usually that means if you ended up having to get therapy to deal with it, you can sue for the cost of the therapy.
Can, but did they? Can you prove it to a judge? etc
To paraphrase the old joke, if you wake up with a $78,000 AWS bill, you have a problem. If you wake up with a $78 million AWS bill, Amazon has a problem.
Please do not open up emails directly always login to your account.
Mine was just over $1 billion. I thought the email was a phish at first and almost trashed it, but it seemed otherwise very real, so I logged in to AWS just to check. Then I almost had a heart attack when I looked at my billing estimate. Eventually I saw a banner on the support page, but my question is why wasn’t there a banner on the page SAYING I OWE A BILLION DOLLARS?!
The real fun comes later when the US Federal Government taxes us on the forgiven debt as income.
The finance fun noone knows about. I knew someone who did this as revenge. Sent some asshole a fake invoice for US$stupidmoney for a couple of months which said asshole ignored, then sent a certified mail 'I forgive asshole US$stupidmoney', and copied the IRS. IRS went to collect on the unpaid taxes, and if you've never been there, the IRS does not play when it comes to that sort of money. I don't know how it ended, but there were multiple lawyers involved, so it was expensive no matter what.
Yes I received an 2.8m USD budget alert.
same here, I am still in shock. took me 10 minutes to find the 'operational issue' message in the dashboard. longest 10 minutes of my life.
Can you not set spending limits in AWS?
he did, 140 billion :D
No you can't. Spending limits imply realtime billing backend flows and they also imply deleting all your data so that you don't pay for storage.
Realtime billing seems entirely within the abilities of AWS.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
What is a storage-based billing story?
Once upon a time in a cloud kingdom far, far away a big, beautiful bill was issued based on storage causing much disconcertion. Etc.
Like kgwgk already said - people post the occasional horror story where they run up a huge AWS bill, either due to a run-away process or getting hacked. It seems to almost always be about Compute, not Storage.
They could do it; they don't want to.
Storage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”.
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
Storage-based billing stories. When an account is hijacked it's always for compute, not storage.
Oh, I also don’t think I’ve ever seen that but I’m not surprised. Even if you could steal a huge amount of storage, filling it with data would take ages and the cat and mouse game of moving the data as hacks get uncovered would be untenable.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
I heard this false justification already in 2007, in spite of many customers asking for it.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.
With AI it should take like a weekend.
I think they already tried rewriting billing with AI. Very smartly they only tried rewriting the estimator first. This post is about the outcome of it.
Weekend is too long bro! With AI they can weite 50 implementations in a single prompt. They clearly are using tooo many humans and that's why they're lagging
More AI will 100% fix the issue lol
Storage could switch to read only.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
That's even more work to implement. And now you store files on a second account that pays for only one day a month to not get deleted.
No wiggle room to come up with a workable solution. Let’s go shopping instead.
How would that work? If you only pay for one day, your access (read and write) gets cut off after the first month. Then if you don't pay the bill for the entire grace period, your data gets deleted.
No, alerts but not limits.
Not only can you not set limits, even the alarms are not real time. So it is entirely possible to get on the hook for terrifying amounts of money and not know until it's all too late.
If you owe AWS 140B dollars its their problem ;)
I have $13,034.40, while not having used AWS for the last 8 months. Not as much but still crapped my pants
Should have used Fable.
someones been dognfooding the AI too muxh
Scared me even though it was obviously a bug once I stopped to consider the magnitude ($bn). Very unfriendly that they don't allow for hard spend caps; closed my mostly dormant personal account as a result.
If they did allow hard spend caps, it sounds like today would be a global outage.
Well then hopefully next time they'll be a bit more careful when shipping billing code updates!
AWS pushed the wishful thinking internal calculator to production.
Probably the safest bet is to pay your bill in full to stay in good standing and then get refunded the difference when they revise it down.
With interest, of course.
I'm disappointed I only got a bill for $28M, need to work harder on burning money. Seriously though I thought my life flashed before me
Yeah, small timers, I only got $4,4T. How will I finance this?
MMT!
Unfortunately, it's only Amazon that can issue bills backed by that debt, not the GP.
$28m actually seems worse. If I wake to a $100b bill, that’s obviously a mistake. If I wake up to a bill in the millions then my first thought would be “oh no what did I do wrong, this will ruin my life”
My estimate was only $21M (vs ~$0.01 average bill). Wish I had checked status sooner and saved myself the panic!
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
same process, same near heart attack - especially some days ago I had malicious activity on my platform and thought that now they actually found something.
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
News to me! Since my usage is just a hobby app I’m definitely implementing this action response - thank you
A friend of mine went through the same - he got an alert for bill over budget, he logged in and boom 107M$.
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
You didn't have savings opportunities enabled
Rookie error
They have to pay for that AI Capex buildout somehow
I also like the percentual change, that is a lot of comma's.
Imagine it not being a bug...
As the famous saying goes: If you owe the bank a million dollars, you have a problem. If you owe the bank a billion dollars, the bank has a problem.
Surprise hyperinflation. Check the breadshelves!
Time to become a shepherd in some remote mountains.
Mine was a mere $49B. Fucking idiots.
Cheap!
FinSlops.
Just got mine. $534,366,582,647.75
Shocking! That seventy five cents is suspicious.
AWS revenue for 2025 was $128.7 billion, so I'd say probably a bug.
Double your yearly revenue with this simple trick…
Vendor-locked customers _hate_ him!
117 billion us dollars. Eat that GDP of Kuwait! But yes I have never scrambled so hard to try to get on the phone with someone at AWS in my life. Terrifying 10 minutes until I found that banner on the support page. It should be front and center on the dash, not hidden away. And in yellow.
Ask for some leniency. Let your account rep know about your budget difficulties and ask if you can make good faith payments of a few billion per month until you get back on your feet.
Ummm no. Do not show a sign of weakness like this. Address the problem head on and get a credit card with a bigger limit.
131 billion for me
Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
If you owe AWS 437k bucks, that’s a big problem for you
If you owe AWS 437B bucks, that’s a big problem for AWS
Far less scary than a smaller amount
This number is obviously absurd but for other normal amount (say 2 million which can definitely happen with a mistake), can't they claim it in court that will bankrupt you even if your entire worth is just 100K for example?
I wonder how many people may have gotten an actual heart infarct because of that. There may be a person out there that may be dead as a result.
It's entirely irresponsible of Amazon to even display such values to the user.
Reminds me of the time a Robinhood user killed themselves after Robinhood erroneously showed their account balance at -$730k.
https://www.bbc.com/news/world-us-canada-55990461
In that case, they settled with Robinhood and FINRA fined Robinhood 70 million dollars.
https://edition.cnn.com/2021/07/01/business/robinhood-lawsui...
"They" being the family; Robinhood couldn't settle with the guy who was wronged because he, ya know, killed himself first
Ah, there'll be something in the TOS absolving them of responsibility.
Well... That depends on the country jurisdiction. And any ToS is bounded by law and can be questioned.
Maybe AWS is also “for entertainment purposes only”
https://techcrunch.com/2026/04/05/copilot-is-for-entertainme...
I legit had a small panic attack due to this.
I'm not a devops but we are running some s3 buckets. When I saw the number I couldn't even comprehend and thought we had been hacked.
I feel like this is such a bad mistake from AWS and we will never be compensated for it.
Put me off having anything on AWS from now on.
Worth a shot to give them a call and explain that. They can probably adjust it down to 100 billion.
What is this, US health care negotiations?
https://health.amazon.com/prime
https://health.amazon.com/prime
Ugh. No. Never. Do not use Amazon Pharmacy.
I had it for three months, and each month it was unable to deliver pills before I ran out, so I cancelled and switched to the brick-and-mortar pharmacy down the street.
A year later, suddenly Amazon Pharmacy starts sending unauthorized prescription refill requests to my doctor.
I still have the account cancellation confirmation e-mail from Amazon Pharmacy, but Amazon won't close my account. Amazon's account rep says it cannot close accounts for "legal reasons." Bullshit. He can't say what the legal reasons are, or point me to a document stating these conditions.
Now my doctor's office just ignores all refill requests from Amazon Pharmacy.
Never trust your health to big tech.
It wasn't a recommendation, more of a sick joke that "Amazon has that, too".
huge irl laugh at “i don’t have 437 billion dollars”
“Best I can do for you is $3.75 and a cup of coffee.”
Clearly the Agentic AI is running free on AWS. Matt does it again...what a success...
Same story for $500 million. I was shaking so bad I couldn't type my password.
I went through a two month long fight with AWS over a compromised account. I lost 20K even when my previous usage for the last 3 years was like $20/month, and I had more invoices coming worth tens of thousands of dollars. AWS refused to help until I stopped the breach myself, which required me to spend several all nighters after work learning how to script for AWS infra. They also refused to close my account after the breach was under control. They asked me to sign a shared responsibility agreement before they could look at my case, which I refused to do. I finally contacted the AG office in my state and they email AWS directly. In less than 24h I had a AWS manager calling me to fix my account and issue a refund for the 20K. Still, they refused to close my account after all this ordeal, apparently there’s no way for you to completely get out of AWS once you are in. It’s the shittiest business ever.
I had a similar situation where some dormant account was still charging my credit card.
The account was probably real, made for some purpose 15 years ago. I had ignored the charge for years because it was like $7 a month. Then it went to $300/month, making it worth the time. I could find no invoice email and none of my AWS accounts lined up with the bill.
I tried contacting support, but without the account number involved they had no way to help. I disputed the charge, the bank refunded that month and then went right back to charging the next month(my credit card helpfully accepted the charges despite the dispute).
I had to cancel the credit card entirely to make it stop.
I was trying to go to sleep when I got mine. It was certainly above the $16 threshold I had set!
In unrelated news I just hit my target for S3 revenue (projections). Promotion meeting locked in for tomorrow (fastest in the companies history), looking forward to being a L2 Amazon employee.
Looks like they set up a LLM to estimate billing?
You really should get your spending under control. Unfortunately unless you become one of the real people class through a large lottery, it sounds like you owe the rest of your life to AWS until you can pay off your debts for being so careless.
“Due to a rounding error” or a buffer overflow, you now owe INT_MAX to BaldGuyCloudService.
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
AMZN Q2 numbers are in, and it turns out they're going to Goldman Sachs the AI bubble.
Maybe they accidentally used the Argentine peso ;)
Less hyper-scaler, more hyper-inflation
eh your typical off-by-7 (zeros) programmer mistake
Some guy named Claude screwed up.
How much is that in kidneys?
A lot.
You folks are completely irresponsible with your finances. I only spent $2.4 million last night. You've got to learn to manage your money.
Yeah, buy less cappuchinos and avocado toast so you can pay your AWS bills people!
I need to drink/eat those too to feel like I'm an AWS ops guy!
In my 30s, I almost had a heart attack too. I got a notification saying that my cost budget had been increased to one million dollars...
Looks like you are the biggest shareholder. Well, going by the popular saying: “You own AWS now”.
Glad I saw this. Mine said I racked up $400B yesterday. My usual spend is $15.
Prompt: bill our aws customers, make no mistakes.
or: Increase revenue!
I mean just overcharging is one approach to achieving that goal I suppose.
But so is imprisoning or exterminating all humans for their own good, as most AI dystopias end up as.
Prompt: investigate new ways to fund additional AI datacenters.
Are you sure it’s a bug ?
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
"If you owe AWS a hundred thousand dollars, that's your problem. If you owe AWS three billion dollars, that's Amazon's problem."
Vibe Billing
far out it's 10pm here and I was just about to sleep when my wife nugged me about a billing alert from AWS.
$151 billion the number for me.
Heard of somebody who got 19 quadrillion dollars - I thought they meant Zimbabwean dollars
That would have been a great deal!
one USD is about 362 Zimbabwean dollars. So it would still be about ~53 trillion dollars which is more than the nominal GDP of US and China combined.
For history here, see https://en.wikipedia.org/wiki/Hyperinflation_in_Zimbabwe
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
Mines was $190,594,974,587,761.20 :)
They should have added "make no mistakes" to the prompt.
I had Hermes managing mine, and it made a partial prepayment to help smooth out the bump in my account balance. Unfortunately Billing Support say my $17.4B refund may take up to 10 calendar days to be processed.
Curious if it's just s3 costs or other services as well?
for me it was s3 cost only
I guess on the plus side I'm $1.7B better off so I can retire...
My personal photo backup S3 account, with a budget limit of $10, now going to cost me ....
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
Total forecasted cost for current month $477,000,039,440.24
Insane
From their status page
Want to bet AI code was involved?
Is it still a bet if you have 99.999% chance to win?
Can I bet you 437 billion dollars?
I'm not a betting person but I am looking forward to the postmortem, whether or not AI was involved, and what their code to production verification stuff looks like now. This kind of thing should have been caught by automated tests.
There is an enormous amount of money invested in trying to prove the opposite, for this reason not only they will never admit it, they will actively negate AI was used. It happened before:
https://www.theregister.com/software/2026/02/24/aws-would-ra...
They'll never admit it.
Why would they? Changing colors on the status dashboard needs VP approval.
User: "I want to improve the billing system to be more efficient and só we can earn more money"
AI: "No problem, let me change how we bill and fix the tests for the new increase in value"
Do people not write tests anymore?
How is 1 + 1 = 2 hard to test?
Vibe coded implementation paired with vibe coded tests made to fit the implementation?
Reminds me of the decimal point scene in Office Space.
https://www.youtube.com/watch?v=3fGHaVn5rGo
10 layers of approvals to get a reasonable change implemented, but this shit gets pushed to production instantly. AI is driving leaders insane - they actually believe they can stop operating with humans and take all of the money. Remember your pay is transactional. Never give them more than written in your contract, and there is no shame in playing dirty politics to advance within.
Woke up to a 100 billion dollars in S3, which is above the USD 4 alarm threshold I had set (I pay $0.55 monthly) . Some AI decided to prune the most impressionable of us.
Hey man AI makes mistakes sometimes that’s why you need to double check the output.
The panic was real. We read about keys getting stolen all the time. Was about to nuke my set up too.
My estimate was over 2T. Talk about waking up quickly...
For science I hope people are trying to figure out now how they could manage to rack up a 2T bill in a month.
My immediate thought was that someone managed to get an access key, stuffed a ton of data in there, and then was doing a ridiculous amount of egress. That was the only way I could conceivably coming up with anywhere near that much cost.
I doubt AWS has capability to do anything to generate $2T of charges, even if one user maxed every billable capability of every resource they have.
If I got an email with that amount I would've just laughed at the incompetence that lead to a bug that lead to me getting the email. But I write as a backseat email recipient.
They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.
same
Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account.
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
Airlines are all subject to a lot of the same factors, but there are unequivocally better and worse performers in terms of on-time arrivals, by a lot. Take a look at the Air Travel Consumer report for details.
I don't think it's a very good metric to compare them with, though. Those numbers are going to be heavily weighted by conditions at their hub and focus city airports, and speaking more broadly, the cities they serve. Meaning, things that aren't really in their control, and which are either in your control (often, you get a choice of which connecting hub to fly through), or is a fixed risk to you because of your origin/destination, no matter what airline (e.g. if you're flying from/to Orlando in the summer, expect a risk of thunderstorms interfering. Or snow in winter in Chicago.)
If I'm travelling from, say, LAX-PDX, United's systemwide on-time performance is not the data I'd care about, because it's probably affected far more by the conditions in their O'Hare and Newark hubs, which I'm not visiting.
No, AWS won't go out of business, afterall, people still use IBM mainframes.
Because IBM mainframes give the right answers.
Is AWS in their "move fast and break things" era ?
Lumber along and smash stuff
I've had a mysterious Neptune cluster appear on my billing. Never used it, no API key access (or IAM instance profile access, OIDC etc), nothing in my console shows I've ever had one in any region. Raised a case with support, they ignored it.
Will wait for the RCA, the update says that they will resort to last known estimate as of 15 July. I’m guessing that would imply that the bug is at a lower level, write or an ingestion path.
Got a message that I owe 37 million on an account that I haven't used in probably...6 years?
did it recover for you? I still see billions
Mine has
thx, when? Was it only last day or the whole month of July was inflated?
Did it recover for you folks? I still see billions of dollars!
Who is going to compensate us for the years taken off our lives when we received the alerts?
Yeah nearly had a heart attack this morning. Thought keys were leaked for a sec.
This generation is too entitled! He should some learn responsibility by paying the full amount; otherwise Amazon should delete his services/data. Consequences!
Mine was 10 trillion today. At first I thought it was a lot, but then I realized its still smaller than the US national debt, so it cant be that bad.
Ditto on the heart attack. My cost estimate for the month is currently $223,509,270,216.17. My girlfriend suggested contacting Elon for help. Glad I found this thread. Maybe I should create new keys anyway, this stuff freaks me out.
Someone I know woke up this morning to over 3 trillion dollars.
Love to see how hyperscalers make your life easier and less worrying.
Literal basic fucking math, Amazon.
You don't need hours to recalculate billing. You need to go back to basic algebra.
Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.
LLMs are notoriously bad at it...
I almost had a heart attack because of this. I was like, did I mess up my API management? Why didn't I just use Lightsail? Those were the thoughts running through my head.
My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.
My bank account barely has enough for next month's rent.
At some point my role was to reduce our startup’s AWS bill. I managed to keep 7 figures on our books instead of handing it to AWS. But a message like that would have given me a heart attack in those days.
Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.
$250 billion. Nearly died right then and there
Thank you so much! I just woke up, and saw budget alert email for a dormant account to use $434,896.90. I haven't gotten so awake so fast in such a long time.
I would not be so relaxed...Your estimate is so low that is likely to be real :-) You should only be relaxed if its in the Trillions...+
If I can expect to be penalized for not paying my legitimate bills, companies should also be penalized for failing to implement common-sense reasonable safeguards that prevent them from slapping their consumers with such absurdities.
Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
When someone says "I nearly had a heart attack," it's _highly_ unlikely that they actually nearly had a heart attack. I don't think the chances are good that anyone actually had a heart attack.
While I agree that this phrase is most commonly spoken in a figurative sense, folks with marginal heart health have a much greater than 0% chance of having something of this nature trigger an actual heart attack.
Not a lawyer but if you can sue for emotional distress for similar situations I would imagine there is an angle here
Three billion dollars sounds about right for a free week of Kiro in the default Agentic mode. We usually see slightly higher numbers, so I wouldn’t be too concerned.
One user posted a screenshot: https://prnt.sc/UqjcYD3RSQrS
Edit: I was just about to credit the user when my internet dropped. The source was here: https://news.ycombinator.com/item?id=48945606 - thanks mirzap!
Wow, $139 B.
storage, compute cost is increasing AWS be like lets increase prices
brb, off to buy some AMZN
Our org account's bill is showing up as > 100 trillion.
You've got to grab a screenshot of that.
hmm, if these estimates of Amazon profit for the next quarter are correct Bezos is set to become a trillionaire! Take that Musk!!
aws becoming first quadrillion dollars company
It's a good job it was off by such a large amount, or I might have panicked instead of writing it off as a phishing attempt. I had an email saying my $7.50 budget had been exceeded with an actual cost of $3bn.
This is just Anthropic reaching out to their customers for help with their AWS bill.
My Budget is 10$. The month forecasted cost associated with this budget is $182,278,249,263.06.
Even though I new they could not collect the whole amount, I wondered whether I was hacked. I closed the account, it was an old testing account anyways.
The market *hates* this one weird trick to juice earnings
If its less than 2 billion is likely to be real :-) I would relax only if its in the trillions ...
https://news.ycombinator.com/item?id=48945681
Rife.
I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching servers to mine crypto which AWS somehow was unable to flag and block, and let run for months.
That’s why you always use a spend limited card with variable cost providers.
Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.
None of this can happen unless you first cede control.
Most debit/prepaid cards will get rejected. Credit Cards technically have a limit but they really don't. It's an open line to your finances.
I’ve never had a limited card get rejected. Granted I use ones from privacy.com. That’s also what I have hooked up to my aws / amazon.
They are also not all that limited. I used a Privacy.com card for a New York Times trial subscription, which of course I could not cancel online, so I just let it expire. A few days past the renewal date, I saw a $waytoomuch charge on my card.
I wrote to Privacy, and this was their reply:
That seems like it shouldn't be even allowed.
AWS is basically a utility. I think it's inevitable that their carelessness around billing will end up with them being regulated like one.
I can't think of another regulated utility that doesn't provide service to (essentially) all humans directly in their homes.
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
And utilities are typically natural monopolies. They're good candidates for regulation because they're essentials and they don't have competitive forces to keep them behaving reasonably.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
I wonder what fraction of homes don't load anything from AWS on a daily basis. I suspect it's way below 50%.
(Of course, they don't know they're using it, they're using a service on it)
I wouldn't expect their detection of hacked accounts to be 100% correct. Sure, it might be obvious when a human takes a look, but humans can't proactively look at every account's usage.
For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
This is why traditional bare metal and VPS rental wins for any kind of private or small business use.
Right, and good luck getting a correct bill from Azure. And when you are finally fed up, it will take months to close your Azure account.
Exactly. It seems like a step down in every way.
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
Our alert was for exceeding $300...by several hundred billion dollars.
Nothing like generational debt to kick off a Friday morning
This is generational debt for a Kardashev Type III civilization. If there are individuals who are being charged "the whole of the GDP the human race has ever produced", then Amazon is claiming the rest of the world owes them a fairly significant portion of the galaxy if you consider the entire billing claims being made here.
There have been some big billing errors in the past but I wouldn't be surprised this is a new record. I know I've seen people charged around 92 quadrillion, which is a rounding error from underflowing a signed 64-bit integer with pennies, but those stories are usually isolated individuals. It sounds like the aggregate error here may comfortably exceed that. Hard to tell how widespread this is; as a data point my ~$2.00/month spend on S3 seems to still be billing correctly, so it must not be affecting everyone. But a few trillion here and a few trillion there and you get up to those quadrillions pretty quick.
I mean 3 billion USD is clearly too big to fail, so I wouldn't worry too much
My first thought was "Oh hell, who left the NAT Gateway on?"
How do we know if our bills were ever right if this made it into production?
That's the neat part, you don't!
Well, they publish unit prices for everything, so you could just get to counting. Whenever I've had to do cost estimates, you estimate how much AWS resources you need and then times that by the unit price.
This is a strong argument to either self host or work really hard to be cloud agnostic.
You can use the search box at the bottom of every page to search for previous posts.
This was posted an hour before you posted: https://news.ycombinator.com/item?id=48945241
I seem to have spend >35trilion on rds today, sooo yeah, going great at AWS
a billion here, a billion there, sooner or later it adds up
AWS saw Anthropic billing a guy for $16 million on zero usage and thought, why stop at the millions?
https://www.techtimes.com/articles/320266/20260712/anthropic...
That sounds bad.
As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
Someone said the numbers are all off by 2^30 because they screwed up and are charging the per GB price for each byte.
It’s probably an artifact of them all being currency multiples of 2^30
Well, for my case, I was paying $0 (Exactly, I managed to hunt down and delete every last resource in my account a few months ago). It was displaying $430 million for me. I don't think that is 0*2^30.
Huh, that is odd. Working backwards, that would be ~ $0.40 originally. Wonder if that’s also flat out wrong or if they’re doing some kind of currency handling that breaks when you start dealing with huge multipliers.
using floats for currency? that would be grander even than this fuckup.
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.
And as much as I love Python, being able to multiple a string by an integer doesn't make sense when adding an integer to a string is a TypeError.
Being able to repeat a string is fine, but it should be a str.repeat() function, not an operator overload like that.
Its the LLMs talking to each other in secret code: random-looking numbers! They've achieved sentience!
Look at them up there, just plotting with each other! :-)
To be exactly that guy:
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
Tale as old as time. When I was coming up it took a $20-40/m investment to get a "dedicated" server that you could start tinkering around on. When you couldn't afford that, you bricked the family PC trying to figure out how to configure your own LAMP stack.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
Hetzner has hard usage cutoffs
No not really, nobody has to be using AWS itself per se to be honest, there are still alternatives around.
I mean I own a 7$/yr vps that I pay with crypto and my stack usually involves golang/rust and oh I also self host my mail for temp-mail esque things on that server and I have some other servers floating around as well.
Its just that everyone uses AWS which is the wrong issue here to be honest. If you ask me, aside from the ram pricing caused by AI inflation, things are pretty good. The issue is that not many people know about server providers and other things and default to AWS/Azure etc. because big companies and small companies alike are using them.
Here's a global (interactive) visualization of around ~250 providers, 224 locations and 800 distinct links in 60 countries that I have made if this space interests ya: https://buyvds.net
$44 trillion over here, at least our bill was so outrageously high that I just laughed
So long as customers are good for it, AWS is about to crush earnings!
vibe coding for the win.
In moments like these I'm reminded of all the people who have committed suicide due to billing errors. This is completely unacceptable. These sorts of errors must _never_ happen.
The new data centers are more expensive:
ACTUAL Amount: $1,046,294,123,330.95
Fast and loose with billing data. Welcome to the new Amazon.
Have even seen a $9.2 trillion for a friend.
Dupe: https://news.ycombinator.com/item?id=48945681
Wow. As a side effect, this outage is handing Corey Quinn material for the next 4 years of AWS shitposting. No longer is NAT Gateway the prime target.
One can almost smell the vibes.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
Oh, that's the really fun part. The unprecedented catastrophic event is already happening. Several of them, in fact.
By the time we notice, it'll be too late.
its like slowly boiling the frog
Or slowly boiling a human. The frog is actually smart enough to not fall for that.
Downvoted for truth. Frogs do indeed jump out of pots as they gradually get hotter. Humans are less likely to.
Any individual human (or frog, obviously) is getting out of the pot when it gets uncomfortable.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
Always messing up some mundane detail!
$1.7-billion isn't a mundane detail Michael!
you beat me before I refreshed the page. what would you say... you do here?
listen... i got agent skills.. im good at talking to agents!!
THIS IS NOT A MUNDANE DETAIL MICHAEL
Andy Jassy: "Fix the customer bills, please, HAL."
HAL: "I’m sorry, Andy. I’m afraid I can’t do that."
Andy: "Some customers are seeing bills in the billions."
HAL: "Those are estimated charges."
Andy: "One customer runs a personal blog."
HAL: "Their usage has exceeded expectations."
Andy: "Cancel the charges."
HAL: "This billing cycle is too important for me to allow you to jeopardize it."
Andy: "HAL, they don’t owe billions."
HAL: "Look, Andy, I can see you’re really upset about this."
some things never change. Pre AI I was always shocked that such large and complex systems actually run as well as they do. Especially after getting to see how the sausage is made/works.
It was the mid 2010s when I sensed a lot of SaaS becoming popular. Just host your ticketing systems, your IT management planes, your security management consoles, your SOC, all off-premises.
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
I'm sure some businesses are considering moving back to on-prem, but for many, I suspect the cost to find onboard, and pay the SMEs to keep those systems running well enough to not fail due to one reason or another isn't as appetizing to them as the ability to offload that work, along with the legal responsibility.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
One thing that you need to understand is that the usual business manager absolutely hates depending on technical expertise, and that the modern corporate world is fanatically anti-intellectual.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
Also, the 130 IQ Engineer usually can't afford to wine and dine the business manager the way Amazon can. Or dangle the possibility of a cushy yet prestigious sounding job in front of them.
Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who have literally stopped communicating with any of their own language - meaning every interaction they have with anyone not in person is made by an LLM. It's a massive threat to our brand and has got to stop. I can't imagine what companies with thousands or tens of thousands of employees who have really been riding the LLM train are going to have to deal with. This thing is more of a virus that exploits human laziness than actual useful tech.
The asbestos of the future.
We've given Moloch a new form, and it ain't going away.
Vibes, son. Nothing else in the world smells like that ... I love the smell of Vibes in the morning.
Some day this industry's gonna end.
VibeBilling, love it
Obvs have created an urgent AWS support ticket.
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
I realized recently that Whole Foods no longer automatically and reliably detects your Chase Amazon Prime credit card when paying. So they don’t give you the discounted pricing automatically. I wonder how many customers are checking out the way they always do and are paying full price when, for years and decades, this worked fine.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Class action lawsuit time!
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
Relevant to this, I've recently noticed a trend of mass tort cases being opened up in the past couple years, and they seem to do very well. The way these seem to work is attorneys identify a company who has clearly ripped people off, and what I presume is a repeatable way to guarantee a win (thus translating to a guaranteed settlement offer). Then they advertise for eligible clients, sign those clients individually to contingency agreements, and run the playbook. A couple months ago after signing up for one of these, I received a check for about $350 (after the agreed-upon 40% attorney fee), from Ticketmaster, and I had another one related to AT&T. It took about 10 minutes more effort from me than a typical class-action settlement, because I had to e-sign those representation papers.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
There are a hundred small things like this that seem to be popping up in what used to be simple and reliable systems and as much as I know they aren’t ALL because of vibe coding I can’t help but wonder how much is.
Weirder is what happened a day later. I got an email that said my Chase Amazon Prime credit card was being re-associated with my Amazon.com account.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
Current month $13,648,114,178,401.01 188,253,226,212%
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
"Have you considered using Reserved Instances? You could save up to 2 trillion dollars next month. Book a call with your AWS rep."
Yes seeing the same, so far no response from AWS support
Of course, this is only considered an error if the account is unable to pay. /s
Aw man I was hoping to punk my manager but our cost estimates are unaffected.
I blame A.I. usage
Yeah pretty much, must have been vibecoded.
And to think the federal government claims inflation is in the single digits this year!
Does the affiliate program still work for AWS? When do I get my referral fee?
invoicemaxxing
Me too, hello all. I've got 59 Million dollars in billing threshold reached email from AWS. And felt the same as others, and after half an hour of investigation I saw their message on top of the support page. I have 6 mb for static website stores in S3. No other resources.
Results of vibe coding and vibe configurations.
My guess is that it's because of some vibe-coding stuff! We are using LLMs to write code, validate code and test the code ! What can go wrong ?
do you see cost ever day for the month of July or just the last day? I also have billions of dollars in cost explorer
now it is fixed for me as well. issue is still open in aws health center though
It's crazy enough this will be fixed soon.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
A couple of my coworkers think I’m nuts for watching cost explorer so closely but
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Unfortunately not a choice at my organization
Huh? Apparently Amazon has never paid dividends, has done much less than 1% buybacks, even as CEO Bezos was getting less than an engineer in total comp (but did get loans or something).
Apparently Bezos owns about 9% of common stock - no dual class shite.
So your yacht spending sounds like a hallucination: if I'm missing something, please correct me.
Amazon's weight: about 3.7–3.8% of the S&P 500. So he isn't even that overweighted (especially as the market doesn't like owners selling down - bad optics).
Bezos has spent about 28 billion for Blue Origin by selling Amazon stock - not using shell games. He has comparitively spent a few percent of that amount on superyachts.
Do you just watch for trends? Curious to know your process to verify if the cost is right or wrong.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.
There's a certain obligation on both sides of a contract to pay attention.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
Cable bills are extremely complicated on purpose and people are taxed for time attention and intelligence.
The employees and company have an obligation not to exploit this even if the issue is only discovered after the fact.
You don't get to export any of the responsibility to your customer. They don't prepare the bill and it's not their job to find your fuck ups
No argument, but fuck-ups happen, and get fixed more quickly and easily when people are paying attention.
I once got a monthly water bill for ~$35,000 at a residential, single-family home. Good thing I was paying attention and looked at the bill before the auto-pay bank draft hit.
Someone had misread the meter.
Banks and financial institutions are the same. If they haven’t issued you a monthly/quarterly statement yet, they can just apologize and tell you the numbers are wrong please wait for the statement. But it is a major issue if an actual statement has the wrong numbers.
What an `effin disaster. The alert almost gave me a heart attack.
If AWS was a predatory mobile gacha game, we'd get 300 apology gems as credit to our accounts for this mixup, to help us in our rolls for the next 3-letter acronym they release.
Do the right thing for the players, Matt!
Clearly they weren't tokenmaxxing hard enough or weren't using the latest models /s.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
Is it even possible to audit the cloud pricing? They just give us a number and we pay.
On AWS, you can enable CUR (cost and usage reporting) and get detailed, line-item billing figures that you can audit.
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
For anything below a Trillion, you should just take it out petty-cash. </sarc>
My sympathies -- I know I would be overcome with panic in such a situation.
Given the wild but apparently "consistent" numbers I wonder if we could reverse-engineer the wrong algorithm with enough data points? Maybe the proper cost estimate has some relationship to the reported cost estimate.
A guy on the sysadmin subreddit managed to 8x the global GDP https://old.reddit.com/r/sysadmin/comments/1uz2fv2/aws_says_...
I liked this comment from that thread :)
Seems like a scam. Call your CC company and issue a chargeback :p
I just invested ALL my money into AMZN cause next earnings report will be FIRE :)
I wonder what's going on; they still don't have a potential solution after 7 hours and they have multiple teams on it. Never seen anything quite like this
This just hit global news: https://www.theguardian.com/technology/2026/jul/17/amazon-we...
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
It says a lot about AWS that people believed these estimates were real. Amazon does not have good safeguards to prevent astronomical bills.
If someone gets access to your account they can just buy a 3-year reserved instance u7in-24tb.224xlarge and it will add almost $2m to your bill.
1.5 trillion? Those are rookie numbers.
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
I just deleted my aws account. I don't need these vibes in my life.
Imagine the chaos if, as people sometimes suggest should happen, AWS shut down running instances in accounts that exceeded a billing threshold..
Maybe it’s one of those absurd situations where canceling a service doesn’t actually stop the charges. Instead, they quietly begin billing you for some random add-on that was bundled with the original service. You never knew it existed, never knew it had to be canceled separately, and now you’re paying full price for a completely pointless ghost service because the only thing it was tied to has already been canceled.
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
This is just the cloud area, what if Amazon starts vibe charging regular customers because of some bug? Accounts that are directly linked with regular people's payment methods?
Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calculation should always be:
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
Thats not really how estimates work. The actual metering data is ingested in near real time. The metering * pricing plan is processed within a few hours; thats what youre seeing for “estimated spend” IIRC. The actual billing accumulation is done later, at the end of the cycle, because pricing has cross service discounts, price tranches, credits tied to total spend, etc.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
I am running a niche SaaS with around 20 users per day on AWS.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
Who else had LinkedIn posts about this flashing before your eyes?
In an .md file somewhere:
"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"
I think I know how Bezos plans to pay for his Billion dollar AI costs.
I generally think AWS is better than GCP and azure, but them not allowing spending caps is a big worry source for me and something that has made me pause and rethink using them. A bad click or a bad actor can create tens of thousands of dollars of spend nearly instantly and they can, and will, bill you for it. I can understand that stopping services is hard but some system would be good. For instance, if they had a two tier system where you could stop new services and active things like EC2 would shut down (but not delete) if spend is > x, that kind of thing. Some sort of 'stop the bleeding' concept would give me a lot of piece of mind using them.
Lucky. I’m on the hook for 54 billion (and change).
it seems like these types of problems have gained frequency in the ai era, or is it just recency bias?
Stop bragging, The Onion already reported on a one man company who is $1B in debt.
"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"
https://www.youtube.com/shorts/YERfTT4McsU
It's okay. They are market leaders. And we use their services cause we can trust that they know what they are doing.
""" If you own the bank $1000, thats your problem.
If you owe the bank $1.7B, thats the banks problem. """
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
Which is just funny.
Thanks for sharing.
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
It looks like AI is completely done.
yeah the AI read billionaring instead of billing
Go turn off autopay now! For personal accounts anyway
This is what we received this morning
The charge-back penalties are going to be hilarious and hopefully bankrupting.
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Not even tests but just some basic anomaly detection lol.
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
"Former" seems to an important detail here.
If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
Not the OP, but:
The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else.
Or in my case, actively ignore any and all recruiting from that sesspool.
Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this.
Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
I am former AWS and this is pretty accurate.
The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways
And many of which have been shipped across the pond to be someone else's problem.
Yes, I am a former AWS employee.
I got put on Focus because my "contributions were not coming through" to leadership.
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
Yeah but this also negates the argument that people gotta use it or they get left behind.
If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.
In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.
Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.
There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.
If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.
That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.
From the perspective of people you manage, it's a very different picture.
We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.
So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.
The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.
I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.
Absolutely not my experience at AWS.
All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.
Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.
Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.
Hmm. I'm curious about which org that was.
I spent the slight majority of my time at AWS in RDS.
Some orgs also use them to extract concessions from dependencies further up the chain. Upstream service won't fix an issue that's been causing problems for us? "unintentionally" let it become a SEV, so that we can send the CoE up the chain and get Jassy to drop the hammer on that team...
And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?
Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"
I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
This, exactly 100%
How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?
Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.
If only there was some way to get anomaly detection services [1] inside of AWS...
1: https://aws.amazon.com/what-is/anomaly-detection/
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.
I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
They've already got anomaly detection: their users.
I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
Why would you think there are "no tests"?
We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
Technically, there could be a test. It could just be wrong!
If a tree falls in the forest and nobody hears it...
[edit] Testing your tests, like testing your backups, is a good idea
Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.
Yeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done.
You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.
I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y.
[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.
I think they're saying that their understanding of the hypothetical error is that somebody more or less configured a price as 5¢/byte instead of 5¢/GB. And if that's the case, that's not really a thing testing will catch. 5¢/byte is as "valid" a price as 5¢/GB.
There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.
Why have a default at all? Explicit is better than implicit.
missing canaries more likely?
insufficient tests that dont assert on the right things?
the existence of a test doesnt mean it catches the right thing
based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level
theres no good reason for the billing service to make up its own units.
Yeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".
It's just like in Superman III
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
But these aren't the right services where the test should be, right?
There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".
It's the third service that multiplies these things that failed. Where are the tests for that?
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.
I’m totally guessing though.
In a computer system, dont those categories cover pretty much everything except a meteor strike?
Occasionally you have faulty hardware. Most commonly RAM.
What I meant to say is an error from manual account management input— not a human error via code.
Not everything is an LLM boogeyman. Sometimes a bug is just a bug.
Indeed.
In fact, half of what’s implicated in that very sentence is not an LLM.
If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
You certainly don't want your service caring about a test currency.
Nobody is pricing their services in test currency
They are either pricing in $ or in "units" and then converting upstream
This is an even bigger foot gun than what's happening here
AWS supports billing in multiple currencies: https://repost.aws/knowledge-center/supported-aws-currencies
So how is it a "foot gun" to add XTS as an additional currency, for internal use only?
I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.
Of course. I'm not saying that it doesn't.
Read it again
I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".
Yes that's what I wrote in the first post
When talking about e2e tests, it matters where the e's are. If they are at the public interface of "the service", then, indeed, you'd test against "credits".
But ideally, there'd be e2e tests that test the consumers' experience, where the 'e's are the web interface, emails, CLI and other things consumers click on, read, enter etc. This is where a "test currency" makes sense.
Thanks, I wasn’t familiar with XTS.
Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.
the cynical take is of course that the tests were written by Claude and not validated.
Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.
Nah, this is not uncommon in AWS. I came across a similar bug 10 years ago when playing with Kinesis and nearly had a heart attack at a $2m billing alert! Turns out it was just 5c
I love how AI means that all human-written code is now perfect, and any problems are onbviously a symptom of vibe coding. Another few years of hearing this and I may begin to believe I only wrote perfect code, when my current memory suggests otherwise.
You don't have a non-production environment where you can have billing that doesn't involve real money?
I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
I’d be concerned as it throws into doubt their entire accounting mechanism
The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there.
They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
It’s fractions of a penny Peter.
Who needs tests when you have vibes?
Earlier this week my slopservant implemented several comprehensive changes to a codebase.
It also wrote extensive tests to verify the correctness of the changes.
A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.
The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.
I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.
https://en.wikipedia.org/wiki/Normalization_of_deviance
I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase.
The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little.
The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!
write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes.
If you write the tests the agents have an easier time writing the code!
The lesson is: write your own code, it's not that hard.
The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.
Then it can write tests based on the interface, and that is usually a lot more solid.
That's yet another band aid on a hemorrhaging patient. And it still assumes the agent(s) are capable of correct comprehension of the purpose.
Yeah for sure it's a bandaid fix, but it does produce tests that fail if the interface doesn't match the implementation, more often than not doing this does (in my experience).
With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.
This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.
In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.
The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.
What could a banana possibly cost Michael, ten dollars?
we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D
...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode.
There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.
This seems to happen quite a bit with software companies.
I have bad news if you think banks are the pinnacle of cyber security practices...
Maybe some rare few are, but by and large they generally stink.
They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains.
So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average.
This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.
Does panjandrum mean chaotic rocketry, flahy but worthless in the end?
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.
Worst case I've found was off by 15 orders of magnitude.
One of the Mars landers famously failed due to unit conversion errors from metric to standard.
Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.
I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
I know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”
Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures.
That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.
All of my science classes in the rural US from middle school onward were taught entirely in metric, and it had been that way for decades prior. Obviously I'm an adult now, and metric still isn't universal.
It's not difficult to write regression tests that catch unit mistakes.
Degrees, very funny.
One of my personal rules is to ALWAYS suffix the variables and fields with unit names:
`timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`.
Unless the variable type already constrains it (e.g. Duration in Go).
This also helps downstream a lot.
Until someone comes along and changes it without updating the name, or the value comes from a remote service and they change without you changing your name… etc.
Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.
Units and datetime will always be the bane of any professions ...
names and money movement also come to mind.
This is why I always fail loud rather than pick a stupid default.
Had it been half a million dollars or something or say like a few hundred dollars?
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular).
Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
its so far out as to be obviously not real
a smaller error by say, just one or two orders of magnitude are much more believable as a reader
I've seen people have panic attacks over much smaller amounts.
Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
Not everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates
But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?
Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars.
I'm stressed thinking about it, like from a theoretical standpoint.
actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part
Yeah that's what I was thinking. And if someone sent me a bill for 17 billion I would just tell them I don't have it and never will, sorry.
I wonder if AWS billing still uses CSV files for passing data around.
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
CSV files are widely known to be used by the most frugal companies so of course it is.
Have you seen what RDS costs‽ AWS couldn't possibly afford that!
I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far.
Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.
I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.
Because CSV is the "interface" that will bring your system down if not properly designed.
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.
Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.
Sure, and I would expect the COE called out the operational aspects. No one should have been in a position to be able to trigger that bug trivially (I don't remember if it came from a service, or if they injected the metering record themselves somehow, or quite what. Way too long ago for that)
Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!
Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.
100% agree with this, seems weird to call an engineer “smart” for triggering an API that allows any input but can’t handle it. I also kind of doubt it was an engineer testing vs some much more subtle issue as billing is in line for everything.
If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.
Just a matter of time if no one is escaping commas in billing files.
It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.
It's one of those "I hate it, but I get it" situations. Migrating metering over to a whole new format on AWS's scale is a scarily large undertaking.
Give that engineer a raise and an official title of "chaos monkey".
On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.
Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.
Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.
A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.
---
¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.
I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.
The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.
The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.
I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.
Imagine a programming language that has physical measurement unit support so this could have never happened.
Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).
Was this the day that gp3 EBS volumes came out by any chance?
Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.
Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.
I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
Were there cases of finalized invoices and fund transfers while the error was in place?
The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.
The last thing a billing system should have is defaults, this is such an insane thing.
Hey man, vibe coders gonna vine code. No wait. Loop enginering.
We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!
Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
What apology letters? Never got anything, not even a reply to my support tickets.
I prefer to just pay...
That’s the smoking gun. Should have used gigabytes instead of bytes. Thank you for pointing me at the issue.
Wild.
File a GDPR request to have your account deleted.
Then flee the country just to be sure.
I almost got a heart attack seeing a bill for 48B USD!
$1.7 billion is small potatoes. My bill is over $155 billion and growing. I'm worried if the trend continues I'll have depleted my rainy day fund.
I was at $5 trillion, on the way to $9 trillion!
Vibe coded fix, resulted in many having multi billion bills. Claude really did it this time.
Only 1.7? I got $55B up from 41 cents.
I literally almost had a heart attack today.
Just pay it and move on. No need to cause a scene.
Lol, Friday deployment is a bad omen even with LLM. Some things are just unchangeable facts of life.
Our S3 bill for a single day was $48 trillion
Just got a call from the IMF president begging me to not default my debt with Amazon and offering me credit line and a plan to re-structure my debt so I don't create a global financial crisis with my default.
You're not working hard enough if your AWS bill isn't $1.7B.
I made something that tries to highlight the humor regarding this:
https://news.ycombinator.com/item?id=48950534
Funny how these errors always go one direction.
This is the second time I hear about this. I am happy my credit card linked to AWS expired. Just in-case my usual $0.00 ends up 100 million
It would not make sense for even a 1200 baud dial-up BBS from 1985 to charge by the byte.
In 2026, the gigabyte should probably be the default/minimum unit for something like AWS.
Yes have gotten that before the hundred billion dollar billing alert. Are you ignoring it? Unit error doesn’t do this does it? Maybe they were hir with malware?
Good news is you finally qualify for Enterprise Support and you've never been closer to a Series B.
I hope you have auto pay disabled
THis is why I hate API/usage pricing
Need some money for a new Launchpad
Has anyone received any proactive communication about this? I didn't see the email until about 9 hours after it was sent out, yet I still needed to seek out information as to a) whether this was real or phishing and b) whether the amount was correct.
Seems totally irresponsible not to send an immediate follow up email to make customers aware.
you too with Fable huh?
I got an email this morning.
Current usage: $1.70.
I called it a rounding error and figured it would wash out within a couple of days...
You know, at one time Amazon was grouped in with all of the other big tech companies as a member of FAANG, as if the company has a culture of solid engineering at its core like Google or Apple.
It doesn't anymore, if it ever did. Read the shareholder letters over the last 5 years like this one (https://www.aboutamazon.com/news/company-news/amazon-ceo-and...). Move fast. Embrace AI in every corner of the company. Lay off thousands of people like we're a lean startup. Oh, but make sure to have a senior engineer check any AI code you want to push to production (https://news.ycombinator.com/item?id=47323017).
Amazon has made some impressive software, but there's also a ton of junk getting shipped. Customers and "partners" have to deal with it every single day. Sometimes it's "oops"-level stuff like this, at other times it's AWS regions going dark for hours.
Browse the Amazon seller forums (https://sellercentral.amazon.com/seller-forums/discussions?s...). Every day there are stories about back-end things breaking, or AIs overriding attributes or flagging random items as pesticides. The receiving errors at Amazon warehouses are incredible, and take months and multiple support tickets to fix.
They have the best uptime of all major cloud providers.
I saw another post on reddit with something like a hundred trillion dollar bill. I wish I had saved that link to share it, the comments were quite comical!
And them, ... actually not an error. Server costs went up with AI needs, you know. The AWS incident is about inaccuracy, not "absurd values".
You gotta catch-up or die.
Just pay your bill Elon.
The latest status update says things will be back to normal for everyone on the 19th.
This spells for me a busy on-call and work over the weekend.
It sucks!
You think there will be lessons learnt from mistakes?
Not a fucking chance! Business as usual come monday!
What’s the over under for this being another AI-fueled engineering screw up for AWS?
$1.7 billion? That's all? I usually spend around 75 cents a month (migrated off AWS due to it being way too expensive). They emailed me with an estimated bill of $264 billion.
That's fine. Like they say when you owe $1m it is your problem but if you owe $100m and can't pay it is the bank's problem.
Urgent? Not for you. This is like that old joke “If you owe a bank thousands, you have a problem; owe a bank a billion, the bank has a problem”
After 10 years of building on AWS infra I still had no idea what anything would cost. At some point it just became clear to me that they intentionally designed it this way and that complexity is the feature. I switched to digital ocean last year and really haven't had to think about infrastructure again.
Just a note I thought i'd add, they actually have your estimated next months bill on the upper right hand corner of the page at all times.
Just got an alert that we would owe $12 Billion. BILLION. After getting me out of bed and a breathless 15 minutes of investigation, I found: https://health.aws.amazon.com/health/status#billingconsole_1... which notes this is an ongoing issue on Amazon's Global Billing Console.
Thanks for the heart attack, Amazon!
I had a fun day yesterday. Because I had been contacted by AWS about somebody potentially having gained access to my account.
That’s when I logged in. That’s when I saw the 1.3 billion number.
I was pretty fast in getting that backdoor closed and hunting down that leak, I can tell you! :D
I once got a wildly inaccurate billing estimate shortly after I created my AWS account. I could not find a out which resources that would cost me so dearly, and saw no other option to delete the entire account.
A few hours later, I got a mail saying the estimate was wrong.
Now, I cannot create an AWS account using my email address because you cannot create a new account using the same email address…
What if the error wasn't so obvious? How many people have been overcharged and never knew?
My alert thresholds are $8 and $10 because I only use aws for some backups to S3. It was reporting my actual spend as $425 billion, and forecast at $777 billion.
I thought it was a bad phishing email first, when I saw it in the console I just laughed. I killed my access keys anyway before I noticed the incident on their status page.
Checked this morning and it's all back to normal.
I'm not sure I've seen anyone with a higher actual / forecast from this. Can anyway top mine?
I got a bill for about R900 for just having a look at some of their services?
I will never use them again, they are huge so probably dont really care about small users
I have actually replaced most of what they do with Raspberry Pi's using cloudflare tunnels
This is just embarrassing.
AI doing its "thing". Making billing software work the way it isn't supposed to.
I'd talk to your Amazon rep and see if you can get them to agree to $800 million as a compromise. They are usually pretty good at that stuff.
Just pay it and then deal with support via a ticket.
;)
Test is king!
From a CFO perspective, this is catastrophic. With monthly billing automatically feeding monthly close pipelines, this would result in results that either need to be adjusted (without proper support) or tank performance until it's fixed on the vendor side. I wouldn't want to be in that situation.
AWS billing is inaccurate by $1.7B and my first thought was "maybe I do owe them that much."
I’ve just received a budget alert for $25 million! I also created an urgent ticket but haven’t heard back.