I don't really care about percentages, either. But for some industries, the difference between "two 9s" and "five 9s" can be millions of dollars, so that's why they're published that way to the customer.
What's the point here? That everyone should use the n-nines notation? Sure. However, companies have no interest in doing anything that makes them look worse.
Also, is anyone else getting the bitter taste of AI writing from this page?
Services can have a 50% uptime (or a 50% downtime if you prefer) as long as it's the time when I need it to be up (or don't need it.)
Which is to say that the significance of downtime depends on the user. Talking about nines only makes sense internally when you are evaluating your infrastructure and operations. It doesn't tell you squat about impact to your customer.
Right. And there's nothing wrong with having regular scheduled downtime. That's obviously not appropriate for most massive scale global cloud services, but it's great for services that know their usage patterns well.
GitHub Actions: 12 hours affected in the last 30 days (98.31% uptime).
This is trying to shine the most favorable possible light onto a deteriorating situation. It doesn't take away from the fact that most businesses have measurable missed revenue in downtime. Customers that shop somewhere else, ads that were never severed, leads that grew a little colder. 12 hours of downed GitHub results in millions of dollars of lost developer productivity that was externalized by Microsoft to other companies.
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
One thing I would like to see is how many of those hours are during normal business hours in my country.
Not all hours are created equal when it comes to downtime and my intuition is that most of these 12 landed within my working hours.
In terms of impact that then might mean they were down for 7.5% of the time I needed them, or had business hours uptime of 92.5% which is… both not very good and very disruptive.
On the other hand, downtime at 4AM would be much less impactful even if it happened every day and added up to more overall downtime.
they were down for 7.5% of the time I needed them, or had business hours uptime of 92.5%
You can obviously compute this for a particular customer, but being a global service, it's pretty much guaranteed that someone somewhere experienced the worse of those numbers
You can compute them for the average. In other words, the total customer impact is the number of business-hours of downtime across all customers divided by the total number of business hours of all customers.
This is important because it's quite possible that the downtime is biased toward the times they have the most active users.
Big systems worth their salt already do this as weighted uptime, considering request successes / total requests rather than wall clock uptime as internal SLOs. But these numbers aren't really ever published because it gives away information about your customer base.
"Failed customer interactions" - if you have a way to actually see requests before they hit your datacenter, e.g. some async third party client libraries.
Then again if you’re a business relying on GitHub enterprise you have an SLA and you can and WILL charge GitHub for failing their SLA. Usually there’s a real measurable dollar value tied to that SLA per dollar and it’s not cheap.
What surprises me in particular is that the global API and the GitHub EE API are the same which is a big no no. This is even more surprising given the fact that paying GitHub customers are clearly the minority both in numbers and code velocity.
My assumption would be that GitHub is keeping EE up and the rest of free or pro users just have to suck it up. If that’s not even the case then it’s only a matter of very short time until GitHub will see businesses leave to more reliable competitors
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
They're also completely irrelevant, you as a customer of a service that is down can lose the same amount of money in a 5 minutes outage or 30 days outage, if you were only relying on this service for one operation that took 1 second and had to happen during the time where the outage happened.
Depending on the service in question, no amount of downtime is acceptable, however unrealistic this is.
For example we had a 6 9 (99.9999%) requirement from a customer for any given 3-6 month period. If we violated that, we owed them their money back (baring the outage wasn’t caused by us - I.e our cloud provider shit the bed).
That’s something like 7.5 seconds. For a contract over $1.5M. Am I the only one who thinks that’s outrageous expectations?
EDIT: The web app was for generating SBOMs of static assets.
I worked in realtime trading. No. Not at all outrageous. Quite reasonable actually. If that’s what we agreed and I need you to be reliable I will charge you back for being unreliable. I’m happy to pay top and extra dollar for the SLA but that means it needs to be acted on.
Well, someone on the business side of the house is free to negotiate. Whether engineering learns about the contract before sales has inked a 6-nines availability guarantee varies wildly by the company
If it wasn't prorated anyone who approved the contract needs training and/or firing. If it is prorated, that is generally not a problem. Small outages aren't even worth the effort of trying to get the money back, and if you have a large enough one to make it worthwhile it is likely the prorated refund is still going to be laughably small.
Well, we don't know the application. Six nines is appropriate for some applications. If you're doing PSIP lookups to route 911 calls, that's reasonable. If you're sending out paper mailers, it's not.
But six nines gives you 7.9 seconds a quarter. If you run a multihost system, that translates to ~1s dead host detection and switch and 3-4 switches per quarter. It's acheivable with reliable hardware and reasonable software. Otoh, it's very hard to hit if you need to move traffic to a different location to respond to a no notice location failure (failed automatic transfer switch, all fiber paths severed by construction because the redundant paths were in the same bundle, etc). If you have an out for 'cloud provider failure', that probably covers location failures.
Often times a tight uptime promise like that also comes with maintenance windows. Depending on the application, degraded service or no service may be acceptable within the maintenance window.
The job of the uptime numbers are to look good (and sometimes to meet contractual obligations), more context doesn't make them sound better. Not being understood in layman's terms is a feature.
These companies are happy that you don't know the difference between 99%, 99.9%, and 99.99% and that you think they all sound pretty good.
As more and more things we might consider "platform" move to the cloud, I think it also matters what the service provider means by saying it's up. Just because the servers are alive and responding doesn't mean the platform is really functional.
One vendor in particular we deal with has a powerful feature which we use to a large extent. Unfortunately, that particular feature is all too often not working. The servers are up and the rest of the platform is working, but we need that feature, so if it's down, it doesn't help much that the rest of the platform is up.
These numbers are useful proxies for how likely you are to have your work disrupted outside of your own control.
If you do something 100 times a day against a four-nines service, you can reasonably expect that everything will succeed.
If you do something 10,000 times a day against a two-nines service, you can expect to hit a substantial number of errors during that day, or even have long periods where your work cannot happen at all.
People aren't frustrated with Github because Github has 98% uptime or whatever the specific number is. They're frustrated because it regularly interferes with their ability to work. The 98% number is just a concise way to say it.
Before 9's became a thing, people built systems that required outages, and those outages would happen outside of business hours.
However that's also why some transactions had to complete the next business day after they were registered. Because the whole business was running on offline processing (aka batch processing) that could be interrupted for upgrades, but had to be completed by the start of business the following morning.
My dad did one of those jobs, and my brain has made a bigger deal out of the times I awoke in the middle of the night to find him on the phone at the kitchen table at 2 am dealing with a war room call because an upgrade broke things that needed to be done in 5 hours. It probably only happened 3 times that I know about, and I probably knew about at least half of them, but it felt like it happened twice a year.
Separately from how you present the number, the very concept of "uptime" as a single number is a bit muddy in the context of a distributed system, where different components can be differently available for different users.
Also, 0.1% downtime in the form of a 45-minute outage per month is very different from 0.1% of requests failing in brief bursts. You often see downtime reported as "increased error rates" which is so vague as to be meaningless.
Exactly. If my build and test CI takes several hours and it gets interrupted, it really doesn’t matter how long the interruption was. It impacts me all the added time of realizing it stopped, investigating and confirming why it stopped, triggering another run, and continued monitoring.
If anyone was tracking that time they'd realize running your own build servers is cheaper. But capex is kryptonite to MBAs so you get a shitty unreliable cloud service instead.
There was a push to take the Datadog/whatever dashboard, full of lots of different meters and graphs and numbers, and simplify it all down to a single, easy-to-read-at-a-glance number. Which of course loses all sorts of details on what's actually going on, but it meant that someone who isn't an expert in the system could at least know if something is up or not. That's the trade off, and Goodhart's law applies. The reality is you can't meaningfully boil a complex system down to one simple number. Expertise is still required, and as with all leaky abstractions, to do a good job, you really do need to dig into the systems guts and get your hands dirty and really get to know the details.
I worked at a 4 1/2 nines (99.997) place and it was pretty standard to speak of half-nines at least (although on reflection, perhaps not everyone understood that 0.7 was 1/2 of 0.9 and thought 99.995 was "4 and a half nines") - we went from 3 nines in 2015 to 4 1/2 in 2025 (maybe slide back a little in 2026) - fun task but takes sustained high-level interest in reliability for a long time.
I don't think that people often understand that if reliability doesn't increase before an adoption cycle, the lack of it will limit the success of the adoption cycle.
If I'm tasked with getting a tool to be used at least 2x as much, my first task is to reduce the failures per 1000 runs by 4x. In that way, if adoption increases by 2.5, 3x instead of the the minimum we are looking for, then the number of errors reported per week still goes down instead of up.
We like to think of things as percentages but the moment they start increasing in the time domain everyone gets mad, because they asked for and received the wrong thing.
And that's on top of the fact that when people are 'forced' to use a tool, any errors they experience will be lumped onto the Learned Helplessness theater they've been engaging in to avoid being made to use a tool that is going to make all of our lives easier.
So how about, and I’ll just throw this out there, instead of:
>
> GitHub Actions: 98.31% uptime.
>
> We say something like:
>
> GitHub Actions: 12 hours affected in the last 30 days (98.31% uptime).
The suggested format is equally unhelpful.
You can get 12 hours of downtime by being down once for 12 hours, or 144 times for 5 minutes. The user experience is VERY different in those two cases.
I don't think this is true - people struggle with exponentials. Most people can't intuitively grasp the staggering differences between 2 nines, 4 nines, and 6 nines of availability.
Why should the customer care about how hard 6 nines of availability is? That's the company's problem, and not the customer. If you want to make it more friendly you could just say "10 minutes downtime this week" instead of a percentage.
Wider audience started to use status pages because the service unreliability became so much more noticeable than before and not the other way around. I never had to use a status page for bear blog or protonmail because i never had and issue with it or just i never noticed.
I am now _required_ to consult status page of github, circleci or MS services etc because i need to know why a build is not passing, why i cannot open a repo, why is my work stalling.
Percentages matter, it is just so much more obvious why they matter when it comes down to important pieces of the internet like github. And i highly doubt the number of 12 hours in the last month. MS has been downplaying the issues they have with GH performance for a while now and i don't think it is time to start to believe them yet. Maintaining these pieces of infrastructure is responsibility and a burden.
Overall i would be careful with "nonlinear significance of numbers near 100%" we are talking gh being well into the 90's this year and one number that infra people are also often being reminded about is that "1% is 3.5 days".
Things are tough for gh people and i feel for them but they are not a startup or a underdog of some sort to receive sympathy in that case.
Yeah I'm at a point where I have a bookmark folder of status pages mostly for very large orgs because I've been using those pages relatively regularly. This is not something I felt the need for historically.
Downdetector also doesn't have a motivation to lie.
I've yet to find a status page that wasn't lying about the actual status.
Also 97% up is bullshit for the 3% of people who are offline.
Saucelabs was doubly bad for this because I'm absolutely certain based on traces that they had some sort of demux bug where they would send events from their tunnel to the wrong job. I could see it in the logs that a test timeout was often the cause of an event firing that was looking for something that never happened, because the event immediately preceding it in the script was never fired. Which meant it was either dropped or went somewhere it shouldn't.
Then it stopped one day and there was nothing in their release notes about it. Lies compounded by further lies.
That's just the most memorable example I have. Stuff like this happens all the time and with many services it plays out the same. There's a perverse incentive not to be transparent about problems with the service, so the status pages play down the intensity of the situation.
Microsoft has absolutely gone to shit in the past ~year. Github, Teams, Windows, Azure, Exchange, it's all been fucking trash. Github was running at 80% uptime for a few months, and if they're telling me they're at 98% or whatever now, then they're cooking the books, period. There's no way they're even that reliable.
What's going on at Microsoft? Are they just copy-pasting their github issue reports into copilot and hitting send it without doing code reviews?
Everyone got laid off. Massive cost cutting across all orgs. Engineers are now evaluated on AI usage and pull request frequency, and not bugs fixed or performance improvements.
About four years ago I did interview loops at GCP, Netflix, and Azure at the same time. The latter was a “hiring event” so all my interviewers were from different teams, either managers or TLs.
It was the interview equivalent of the multi-headed dragon meme, where the last one looks absolutely stupid. The contrast was insane, microsoft was an absolute shit show compared to the other two companies in terms of talent, personality, organization and more.
I think that illustrates the author's point quite well. 99% uptime sounds good, but when you think about a 3+ day outage that doesn't sound very good. Imagine Facebook or TikTok being down for 3 days.
Of course most of the time it's not all one outage, but a bunch of short ones. Still, it might communicate the impact better, especially depending on the argument you're trying to win.
I usually tell people you don't need as much reliability as you think.
Three nines reliability is great for most purposes. 8 hours downtime a year.
If your system produces money at a constant rate, it captures 99.9% of the available money. Even two nines or one nine might be pretty good on that basis, when the alternative is spending 2x or 10x as much - let's build another unreliable system with that money that captures some other independent market opportunity.
Poor reliability is a problem where you need to chain many systems together, or where the cost of a single failure is very large compared to a success. Or - as happens commonly because of load - if your periods of unreliability are correlated with periods of maximum opportunity, like an e-commerce site failing on Black Friday or a trading system failing when the market is most busy. But if you don't have one of those cases, evaluate whether investing in reliability is actually worth it to you.
GitHub is an example where two nines of reliability ought to be OK. The argument against it is that it's bad marketing to have an unreliable service, especially one aimed at software engineers. And if GitHub is largely a marketing play by Microsoft anyway (do they really make back its cost in enterprise subscriptions?) then marketing considerations need to drive its reliability.
Actually I think it's better if everything fails at once and everyone can take the day off (thanks, AWS!). Having your CI fail one day and your package repository the next might well cost you two days of productivity.
I fully agree with this take. We need to ensure we get every hour of work from our expensive engineers. This is why we got rid of coffee machines and bathrooms and moved to intravenous caffeine and other fluid drips and catheters. We cannot afford to lose productivity.
One 12 hour outage is different to 12 1 hour outages at 3am which is different to 24 30 minute outages at 4:30pm when you're trying to commit something at the end of the day. Percentage and time are both flawed ways of looking at downtime.
Downtime really matters if it's at a time you need something to be up, and Github is big enough to have users for that to be all the time. That moves the conversation from 'It's down for a few hours a month' to 'Github is failing a significant number of it's users'.
One thing that is I feel missed about uptime percentage when compared to on premise uptime is when the downtime occurs. Its far more impactful if its in the middle of the working day or during the busy period of shopping. A store that goes offline in the middle of black friday or in the run up to Christmas is harmed a lot more than some down time on a Sunday night/Monday morning at 3am.
One thing I have noted over time is a lot of these AWS, Azure et el downtimes is they occur in the middle of everyones day, millions of people are impacted by them. Same with github its getting in the way of work. Whereas when we hosted services on our own equipment the downtime was usually out of main hours. The percentages are in many ways the wrong measure of downtime because hours aren't equal in impact to businesses.
Surely there's a word for this, but I haven't learned it.
The problem with load balancing also shows up in employees mass quitting. It's the same queuing theory problem but I don't know the name of it.
Essentially, when you run a set of 'resources' near capacity, if one fails then the next most highly loaded one also goes past capacity and also fails. The work keeps getting foisted onto other units that also fail.
In retail, the beleaguered employee quits in frustration, and the three other people who can do the same onerous tasks the quitter did now find their jobs have gotten 33% worse with no extra remuneration for the insult, and one of them quits too. The last one or two simply refuse to do the task more than they used to, and the boss can't threaten them with being fired or reported for it because if they quit then the business ends. And still the whole thing goes off the rails.
That's why consistent hashing divides the work of one failed node and spreads it evenly over the remaining nodes, so that a spike is softened as much as possible instead of being concentrated on one or two fallbacks who also immediately fail. It's not that it's a great solution, it's that it's the least bad option amongst some pretty terrible ones.
The solution is to undersubscribe the hardware, which some beancounters hate. But most bean counters don't understand Queuing Theory.
Its not for you, its advertisement, to preemptively avoid both consumer and securities fraud accusations, and for their enterprise client’s IT security review and SLAs
These service status pages have so many other problems, I can't really get excited about this article's point. As a user of a service, I don't really care that an additional 0.09% of uptime is any more or less difficult to achieve than an additional 0.9%, even if you describe it in terms of fractions of time. I only care about two things: what the service status is right now and your service reliability's impact to me over the long term (get out of here with your 30-day crap).
In the electric utility world we have a few IEEE standardized metrics (with appropriately IEEE'd acronyms) for tracking service reliability that I like much better and always wish for when I'm looking at a status page. Pie in the sky stuff for sure, nobody wants to do this analysis and publish the results without a regulator telling them have to, but c'est la vie.
SAIDI - System Average Interruption Duration Index. How many minutes an average customer experienced service interruption in a year. This is the big one I'd want to see on your service status page IMHO. For the power grid, we consider any outage longer than five minutes to be an interruption ("non-momentary outage").
SAIFI - System Average Interruption Frequency Index. How many total periods of interruption occurred for the average customer in a year.
CAIDI - Customer Average Interruption Duration Index. How long it takes service to be restored for the average customer when there is an interruption.
For the "right now" aspect you have probably visited your utility's outage map, but here I would say we do much better than most utilities. The level of detail on the investigation and resolution is often more detailed, and we usually know better than to bother providing much in the way of a concrete estimate for restoration time of a current outage (though this is getting better in the utility space).
We've stopped with our caring about quality and understanding so I don't see why we'd keep caring about reliability and availability. How are you going to pinpoint performance issues when you have no idea how your app works?
Point the AI at your logs, tell it to fix things, rinse and repeat. It's faster than debugging, and fast is good.
I think this idea is objectively good, but practically hazardous. "Four nines" is a meme at this point, and it's a helpful one: a commonly understood gold standard. Few people know that translates to roughly 4 minutes and 19 seconds of downtime per month. Since service quality is degrading, I fear that changing the way we express those uptime might also unintentionally provide convenient cover to reset expectations, e.g. 10 minutes of downtime is the new norm for reliability. Or 30 minutes. I don't think we should yield the Overton Window, so to speak, to this.
But if few people know that 99.9999 == 4m19s, then how is anyone losing out on changes? If you hadn't told me and I didn't have a need to calculate it with certainty, I'd have believed you if you'd said it was an hour a month.
I think the point is that the metric being meaningless right now to most people makes it easier to shift the reliability-Overton-window. 99.9999 vs 98.0 seems not to bad to a layman (I.e. executives), but 4m19s to 12 hours seems pretty intuitively bad. Sure, we may lose some shift in the immediate future, but it's easier for that to continue happening with just percentages, is the point.
It's not like we have to stop showing percentages as well, but time is a much more accessible expression. Right now the lay-ness of company leadership has already allowed the shift to happen pretty markedly.
I’ve worked in systems which at worse have had three nines for years, but I’ve also worked in systems where five nines is a failure.
This attitude of modern tech claiming 98% is good just doesn’t work in the old tech acceptance. We had individual components fail all the time. We’re still looking at a 230ms outage to a branch office last week caused by a power failure combined with a badly plumbed power distribution.
Modern software people don’t consider 230ms to be an outage. Glad they don’t work in electricity.
(The failure we had was only on the services we guarentee at 99.1%, our lowest sla. After that there’s 99.95 and 99.999.
(In reality we reach five nines year after year on even the lowest levels, but there are major concerns like “large bomb in data centre” which could cause some of our less critical units to drop way more than 5 minutes a year.
That's not really an old versus new thing. The water company isn't going to consider 230ms an outage either.
With tech, like, it's crazily difficult to make sure every http request succeeds, so you build in retry, and look at that 230ms doesn't cause real disruption now. Not to excuse 98%, that's awful.
I wrote a blog [1] last week about what we've been doing at Depot to work around GitHub's instability. I wrestled with how to present their downtime in a way that communicates its real impact to users, and I landed on a similar approach as the author. But I think even this doesn't state it strongly enough. Their uptime stat comes from their public status page, so it's conservative by definition. And it's important to remember that their downtime comes almost exclusively during business hours, when people are relying on the platform to get work done.
There are projects like The Missing GitHub Status Page [2] that attempt a more accurate reporting, but taking 98.31% for granted, I think the more impactful framing would be _a floor_ of 1.5 business days lost every month. At 22 business days per month, that can feel more like 7+% downtime.
I saw a User's Group presentation by a guy at Speakeasy, they had made a whole logistic system that looked a lot like a CRM manager but not for customers, for dealing with how flaky Covad was at the time. Covad only had to be less shitty than Qwest (who we called, "Qworst", because if you hadn't dealt with Covad, they were the worst) and they couldn't even manage that.
So if Covad promised to have some line work done by 10 am on Tuesday, the SE techs would get a reminder to verify in had been completed shortly after the deadline so they could call Covad before the customer noticed the install was behind schedule and ask why it hadn't been done by the deadline. Otherwise SE was telling people they'd have Internet in 4 business days and it would be more like 10 - on average.
Around the time Qworst became CenturyLink, Covad bought SpeakEasy, for a net loss of beauty in the world.
Not sure if some one already mentioned, these numbers are rarely for general public
1. These are core to the (service level agreements) SLA's for the enterprise companies when they evaluate and sign the contract (must for government RFPs)
2. The contracts generally have provisions for payback or penalties for missed SLAs
3. When your product depends on any of the products directly then the availability of your product has to take into account the availability of the dependent services. Eg, I can easily sell a SLA of two nines for my service if I am building it on a platform which has SLA of three nines, other way around is always questionable (though possible)
4. All of this is what used to happen may be up until 5 years ago. As of today even with good number of outages, company like AWS has not updated their availability to reflect those outages. Ideally all the companies built on top of AWS infrastructure would have to update their availabilities in a cascading manner but some how everyone decided to skip the beat
5. At this point, calculation of the availability itself has become so opaque that no one understands it anymore so no one questions it either
Many commenters seem to think the author is saying that we should stop caring about uptime, when what he's really saying is that we should be reporting time (i.e. number of hours down in the last month) rather than just reporting an opaque percentage that hides the problem.
I want a good chart not a percentage, because one service being down at 2 am for 3 hours is not the same as another service being down for 45 minutes during my peak hours of productivity.
One advantage of percentages is that you can multiply them if the outages aren't related. If you rely on 10 systems with a 99% uptime each you end up with roughly 90% uptime in total.
After a merger we ended up with 2 operations systems and one of them thought the other one were clowns. But they thought everyone were clowns so it was hard for the rest of us to tell. They never actually apologized for being assholes, but their tune changed when they found they had bitten off more than they could chew later and they had to come hat in hand to a bunch of groups to delegate responsibilities back to them.
So a production outage happens, and we go to look at our runbooks to figure out what to do. The Wiki is also down, so no runbooks. What the actual fuck?
So it turns out the alleged clowns put all of our internal infrastructure onto the same SAN system in different partitions. They got a lot less judgy and the other Ops team got a lot more autonomy after that event.
Anyone with an Operational IQ above room temperature knows that you don't put offline and online resources onto the same hardware. Not only do they have separate duty cycles but also offline services can end up taking out the online services in unexpected ways, in part because they are assumed to be a bit sloppy and so they get less operational scrutiny.
I sped up the average runtime of a batch job by quite a humongous amount by throttling the request rate we made to a service that was used by user-facing services. If several unrelated batch jobs kicked off during the wrong time of day, we would start getting circuit breakers opening everywhere, including occasionally production.
I throttled us to use something like 8 or 10% of the official capacity of the service and no more. But I did it as limiting the number of in-flight requests, so that created back pressure if the servers were under heavy load and sending responses slowly, or let us go faster if the servers were returning results faster than usual. That peak shaving saved the other team a load of grief and let them push some capacity planning work down the roadmap to work on other things. And it dropped our failure rate by more than a factor of six. Low enough that we no longer had to babysit that process, which meant we started using it much, much more.
Reporting absolute instead of percentage values definitely makes it more human. But most of these reports & status pages are held up-to-date manually, by engineers, when incidents happen. I’d bet not even half of the real outages end up reported on there.
“It’s just a few minutes of downtime, nobody noticed, best not draw attention to ourselves by placing something on our status page”
Status pages should be tied into (external/simulated) monitoring to update in near real-time, without hiding incidents. Transparency is the only way. By reporting the _real_ hours of downtime, including the small stuff you think people didn’t notice.
I don't really care about percentages, either. But for some industries, the difference between "two 9s" and "five 9s" can be millions of dollars, so that's why they're published that way to the customer.
Except that customers lean so hard on those extra nines that they create processes where no work can be done when the service is down.
Less reliable products aren't allowed to become bottlenecks in the first place, at least at any B-tier or better businesses.
What's the point here? That everyone should use the n-nines notation? Sure. However, companies have no interest in doing anything that makes them look worse.
Also, is anyone else getting the bitter taste of AI writing from this page?
Nah. Jim is just a decent writer (and historically has been fairly suspicious of AI)
Skip to the last 2 paragraphs
Services can have a 50% uptime (or a 50% downtime if you prefer) as long as it's the time when I need it to be up (or don't need it.)
Which is to say that the significance of downtime depends on the user. Talking about nines only makes sense internally when you are evaluating your infrastructure and operations. It doesn't tell you squat about impact to your customer.
Right. And there's nothing wrong with having regular scheduled downtime. That's obviously not appropriate for most massive scale global cloud services, but it's great for services that know their usage patterns well.
Can we post uptime stats instead? I just axed two of my servers from colocation a couple of days ago. Sad to see them go, six years of FreeBSD.
No since kernel live patching is not universal it promotes bad security practices to maximize the uptime of a single server.
Not to mention being completely useless in the face of FaaS or just autoscaling.
This is trying to shine the most favorable possible light onto a deteriorating situation. It doesn't take away from the fact that most businesses have measurable missed revenue in downtime. Customers that shop somewhere else, ads that were never severed, leads that grew a little colder. 12 hours of downed GitHub results in millions of dollars of lost developer productivity that was externalized by Microsoft to other companies.
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
I don’t see why you think “12 hours affected in the last 30 days (98.31% uptime)”
Is trying to spin anything. It’s making easier to see the impact over the last 30 days. I agree with the article
One thing I would like to see is how many of those hours are during normal business hours in my country.
Not all hours are created equal when it comes to downtime and my intuition is that most of these 12 landed within my working hours.
In terms of impact that then might mean they were down for 7.5% of the time I needed them, or had business hours uptime of 92.5% which is… both not very good and very disruptive.
On the other hand, downtime at 4AM would be much less impactful even if it happened every day and added up to more overall downtime.
You can obviously compute this for a particular customer, but being a global service, it's pretty much guaranteed that someone somewhere experienced the worse of those numbers
You can compute them for the average. In other words, the total customer impact is the number of business-hours of downtime across all customers divided by the total number of business hours of all customers.
This is important because it's quite possible that the downtime is biased toward the times they have the most active users.
Big systems worth their salt already do this as weighted uptime, considering request successes / total requests rather than wall clock uptime as internal SLOs. But these numbers aren't really ever published because it gives away information about your customer base.
https://cloud.google.com/blog/products/gcp/available-or-not-...
"Failed customer interactions" - if you have a way to actually see requests before they hit your datacenter, e.g. some async third party client libraries.
Then again if you’re a business relying on GitHub enterprise you have an SLA and you can and WILL charge GitHub for failing their SLA. Usually there’s a real measurable dollar value tied to that SLA per dollar and it’s not cheap. What surprises me in particular is that the global API and the GitHub EE API are the same which is a big no no. This is even more surprising given the fact that paying GitHub customers are clearly the minority both in numbers and code velocity. My assumption would be that GitHub is keeping EE up and the rest of free or pro users just have to suck it up. If that’s not even the case then it’s only a matter of very short time until GitHub will see businesses leave to more reliable competitors
SLAs don't relly get you meaningful money back, they give you a way to get out of long contracts early and switch to a better vendor.
Favorable spin? The article's point is that "12 hours a month" makes the cost very obvious, and I agree with the article.
They're also completely irrelevant, you as a customer of a service that is down can lose the same amount of money in a 5 minutes outage or 30 days outage, if you were only relying on this service for one operation that took 1 second and had to happen during the time where the outage happened.
Depending on the service in question, no amount of downtime is acceptable, however unrealistic this is.
A lack of ads is an argument for more downtime.
Yeah, this is always fun. Logarithmic graphs of downtime, people.
The 12 hours out of 30 days seems like sugarcoating the issue.
Keep the percentages, and regardless of that - GitHub fix your uptime
I'm also not sure that all downtime is really properly measured now as more and more services are connected and intertwined.
Some measure quite detailled but some just don't summarize the downtime from all providers up and below their own platforms.
`-log10(1 - uptime)`
These metrics tend to be bullshit in contracts.
For example we had a 6 9 (99.9999%) requirement from a customer for any given 3-6 month period. If we violated that, we owed them their money back (baring the outage wasn’t caused by us - I.e our cloud provider shit the bed).
That’s something like 7.5 seconds. For a contract over $1.5M. Am I the only one who thinks that’s outrageous expectations?
EDIT: The web app was for generating SBOMs of static assets.
I worked in realtime trading. No. Not at all outrageous. Quite reasonable actually. If that’s what we agreed and I need you to be reliable I will charge you back for being unreliable. I’m happy to pay top and extra dollar for the SLA but that means it needs to be acted on.
I think in your case it’s reasonable. Also a $1.5M contract is likely chump change in that realm.
This particular case was in cybersecurity- specifically static analysis of assets, for the purpose of providing a SBOM.
It's a contract, you're free to negotiate?
If you agree to those terms knowing it's unrealistic, you're agreeing to give away your service for free.
Well, someone on the business side of the house is free to negotiate. Whether engineering learns about the contract before sales has inked a 6-nines availability guarantee varies wildly by the company
That's a common issue in orgs where sales are incentivized based on size of contract without regard for the profitability of the contract.
See also: selling features that don't exist and cost more to implement and maintain than the contract is worth.
Are you sure it wasn't prorated?
If it wasn't prorated anyone who approved the contract needs training and/or firing. If it is prorated, that is generally not a problem. Small outages aren't even worth the effort of trying to get the money back, and if you have a large enough one to make it worthwhile it is likely the prorated refund is still going to be laughably small.
I think it was prorated but am not 100% sure.
Well, we don't know the application. Six nines is appropriate for some applications. If you're doing PSIP lookups to route 911 calls, that's reasonable. If you're sending out paper mailers, it's not.
But six nines gives you 7.9 seconds a quarter. If you run a multihost system, that translates to ~1s dead host detection and switch and 3-4 switches per quarter. It's acheivable with reliable hardware and reasonable software. Otoh, it's very hard to hit if you need to move traffic to a different location to respond to a no notice location failure (failed automatic transfer switch, all fiber paths severed by construction because the redundant paths were in the same bundle, etc). If you have an out for 'cloud provider failure', that probably covers location failures.
Often times a tight uptime promise like that also comes with maintenance windows. Depending on the application, degraded service or no service may be acceptable within the maintenance window.
The job of the uptime numbers are to look good (and sometimes to meet contractual obligations), more context doesn't make them sound better. Not being understood in layman's terms is a feature.
These companies are happy that you don't know the difference between 99%, 99.9%, and 99.99% and that you think they all sound pretty good.
As more and more things we might consider "platform" move to the cloud, I think it also matters what the service provider means by saying it's up. Just because the servers are alive and responding doesn't mean the platform is really functional.
One vendor in particular we deal with has a powerful feature which we use to a large extent. Unfortunately, that particular feature is all too often not working. The servers are up and the rest of the platform is working, but we need that feature, so if it's down, it doesn't help much that the rest of the platform is up.
These numbers are useful proxies for how likely you are to have your work disrupted outside of your own control.
If you do something 100 times a day against a four-nines service, you can reasonably expect that everything will succeed.
If you do something 10,000 times a day against a two-nines service, you can expect to hit a substantial number of errors during that day, or even have long periods where your work cannot happen at all.
People aren't frustrated with Github because Github has 98% uptime or whatever the specific number is. They're frustrated because it regularly interferes with their ability to work. The 98% number is just a concise way to say it.
But is often wrong itself.
Before 9's became a thing, people built systems that required outages, and those outages would happen outside of business hours.
However that's also why some transactions had to complete the next business day after they were registered. Because the whole business was running on offline processing (aka batch processing) that could be interrupted for upgrades, but had to be completed by the start of business the following morning.
My dad did one of those jobs, and my brain has made a bigger deal out of the times I awoke in the middle of the night to find him on the phone at the kitchen table at 2 am dealing with a war room call because an upgrade broke things that needed to be done in 5 hours. It probably only happened 3 times that I know about, and I probably knew about at least half of them, but it felt like it happened twice a year.
Until there is an industry wide definition of outage, degraded performance, etc then it's all moot
There are regulatory definitions for some industries, like banking or telecommunication (in my country).
Separately from how you present the number, the very concept of "uptime" as a single number is a bit muddy in the context of a distributed system, where different components can be differently available for different users.
Also, 0.1% downtime in the form of a 45-minute outage per month is very different from 0.1% of requests failing in brief bursts. You often see downtime reported as "increased error rates" which is so vague as to be meaningless.
Google's "windowed user-uptime" attempts to deal with this a bit better, by exposing different views of the data instead of trying to condense uptime into a single number: https://www.usenix.org/system/files/nsdi20-paper-hauer.pdf
Exactly. If my build and test CI takes several hours and it gets interrupted, it really doesn’t matter how long the interruption was. It impacts me all the added time of realizing it stopped, investigating and confirming why it stopped, triggering another run, and continued monitoring.
If anyone was tracking that time they'd realize running your own build servers is cheaper. But capex is kryptonite to MBAs so you get a shitty unreliable cloud service instead.
There was a push to take the Datadog/whatever dashboard, full of lots of different meters and graphs and numbers, and simplify it all down to a single, easy-to-read-at-a-glance number. Which of course loses all sorts of details on what's actually going on, but it meant that someone who isn't an expert in the system could at least know if something is up or not. That's the trade off, and Goodhart's law applies. The reality is you can't meaningfully boil a complex system down to one simple number. Expertise is still required, and as with all leaky abstractions, to do a good job, you really do need to dig into the systems guts and get your hands dirty and really get to know the details.
You can use -log10(1-p). On the "nines" it is exactly the number of nines you have:
(Modulo floating point issues of course.)
Which then smoothly covers the entire space:
But good luck getting that standardized.
I worked at a 4 1/2 nines (99.997) place and it was pretty standard to speak of half-nines at least (although on reflection, perhaps not everyone understood that 0.7 was 1/2 of 0.9 and thought 99.995 was "4 and a half nines") - we went from 3 nines in 2015 to 4 1/2 in 2025 (maybe slide back a little in 2026) - fun task but takes sustained high-level interest in reliability for a long time.
I don't think that people often understand that if reliability doesn't increase before an adoption cycle, the lack of it will limit the success of the adoption cycle.
If I'm tasked with getting a tool to be used at least 2x as much, my first task is to reduce the failures per 1000 runs by 4x. In that way, if adoption increases by 2.5, 3x instead of the the minimum we are looking for, then the number of errors reported per week still goes down instead of up.
We like to think of things as percentages but the moment they start increasing in the time domain everyone gets mad, because they asked for and received the wrong thing.
And that's on top of the fact that when people are 'forced' to use a tool, any errors they experience will be lumped onto the Learned Helplessness theater they've been engaging in to avoid being made to use a tool that is going to make all of our lives easier.
The suggested format is equally unhelpful.
You can get 12 hours of downtime by being down once for 12 hours, or 144 times for 5 minutes. The user experience is VERY different in those two cases.
Ultimately the graphs are the most useful format.
Uptime % has the benefit that it's easy to understand.
I don't think this is true - people struggle with exponentials. Most people can't intuitively grasp the staggering differences between 2 nines, 4 nines, and 6 nines of availability.
Why should the customer care about how hard 6 nines of availability is? That's the company's problem, and not the customer. If you want to make it more friendly you could just say "10 minutes downtime this week" instead of a percentage.
Wider audience started to use status pages because the service unreliability became so much more noticeable than before and not the other way around. I never had to use a status page for bear blog or protonmail because i never had and issue with it or just i never noticed.
I am now _required_ to consult status page of github, circleci or MS services etc because i need to know why a build is not passing, why i cannot open a repo, why is my work stalling.
Percentages matter, it is just so much more obvious why they matter when it comes down to important pieces of the internet like github. And i highly doubt the number of 12 hours in the last month. MS has been downplaying the issues they have with GH performance for a while now and i don't think it is time to start to believe them yet. Maintaining these pieces of infrastructure is responsibility and a burden.
Overall i would be careful with "nonlinear significance of numbers near 100%" we are talking gh being well into the 90's this year and one number that infra people are also often being reminded about is that "1% is 3.5 days".
Things are tough for gh people and i feel for them but they are not a startup or a underdog of some sort to receive sympathy in that case.
Yeah I'm at a point where I have a bookmark folder of status pages mostly for very large orgs because I've been using those pages relatively regularly. This is not something I felt the need for historically.
Same ! To such a point I ended up installing a menubar app (like vitalsbar). Never felt the need to have a realtime overview to be able to work...
Nothing beats downdetector.com anyway. Always quicker!
Downdetector also doesn't have a motivation to lie.
I've yet to find a status page that wasn't lying about the actual status.
Also 97% up is bullshit for the 3% of people who are offline.
Saucelabs was doubly bad for this because I'm absolutely certain based on traces that they had some sort of demux bug where they would send events from their tunnel to the wrong job. I could see it in the logs that a test timeout was often the cause of an event firing that was looking for something that never happened, because the event immediately preceding it in the script was never fired. Which meant it was either dropped or went somewhere it shouldn't.
Then it stopped one day and there was nothing in their release notes about it. Lies compounded by further lies.
That's just the most memorable example I have. Stuff like this happens all the time and with many services it plays out the same. There's a perverse incentive not to be transparent about problems with the service, so the status pages play down the intensity of the situation.
Just two weeks ago protonmail had a massive outage related to data center cooling failure.
Microsoft has absolutely gone to shit in the past ~year. Github, Teams, Windows, Azure, Exchange, it's all been fucking trash. Github was running at 80% uptime for a few months, and if they're telling me they're at 98% or whatever now, then they're cooking the books, period. There's no way they're even that reliable.
What's going on at Microsoft? Are they just copy-pasting their github issue reports into copilot and hitting send it without doing code reviews?
Everyone got laid off. Massive cost cutting across all orgs. Engineers are now evaluated on AI usage and pull request frequency, and not bugs fixed or performance improvements.
About four years ago I did interview loops at GCP, Netflix, and Azure at the same time. The latter was a “hiring event” so all my interviewers were from different teams, either managers or TLs.
It was the interview equivalent of the multi-headed dragon meme, where the last one looks absolutely stupid. The contrast was insane, microsoft was an absolute shit show compared to the other two companies in terms of talent, personality, organization and more.
I think that illustrates the author's point quite well. 99% uptime sounds good, but when you think about a 3+ day outage that doesn't sound very good. Imagine Facebook or TikTok being down for 3 days.
Of course most of the time it's not all one outage, but a bunch of short ones. Still, it might communicate the impact better, especially depending on the argument you're trying to win.
I usually tell people you don't need as much reliability as you think.
Three nines reliability is great for most purposes. 8 hours downtime a year.
If your system produces money at a constant rate, it captures 99.9% of the available money. Even two nines or one nine might be pretty good on that basis, when the alternative is spending 2x or 10x as much - let's build another unreliable system with that money that captures some other independent market opportunity.
Poor reliability is a problem where you need to chain many systems together, or where the cost of a single failure is very large compared to a success. Or - as happens commonly because of load - if your periods of unreliability are correlated with periods of maximum opportunity, like an e-commerce site failing on Black Friday or a trading system failing when the market is most busy. But if you don't have one of those cases, evaluate whether investing in reliability is actually worth it to you.
GitHub is an example where two nines of reliability ought to be OK. The argument against it is that it's bad marketing to have an unreliable service, especially one aimed at software engineers. And if GitHub is largely a marketing play by Microsoft anyway (do they really make back its cost in enterprise subscriptions?) then marketing considerations need to drive its reliability.
I'm not sure that having your expensive software engineers not work 2.4 days a year is very palatable
They have other stuff to do that's not bottlenecked on push/pull/CI. 1% github downtime should reduce productivity by a lot less than 1%.
There are n services and n tasks, the chances of overlap are minimal. In most cases people take a much needed break during downtime.
Actually I think it's better if everything fails at once and everyone can take the day off (thanks, AWS!). Having your CI fail one day and your package repository the next might well cost you two days of productivity.
I fully agree with this take. We need to ensure we get every hour of work from our expensive engineers. This is why we got rid of coffee machines and bathrooms and moved to intravenous caffeine and other fluid drips and catheters. We cannot afford to lose productivity.
They already don't work for 130+ days of holidays, weekends, vacations and sick leave. What's another two days?
Probably I already ask them to spend 2+ days attending HR or compliance training or listening to senior management tell them about sales targets.
But the point is, that 1% extra productivity requires the sometimes staggering cost of making the software 10x or 100x more reliable.
One 12 hour outage is different to 12 1 hour outages at 3am which is different to 24 30 minute outages at 4:30pm when you're trying to commit something at the end of the day. Percentage and time are both flawed ways of looking at downtime.
Downtime really matters if it's at a time you need something to be up, and Github is big enough to have users for that to be all the time. That moves the conversation from 'It's down for a few hours a month' to 'Github is failing a significant number of it's users'.
My b
One thing that is I feel missed about uptime percentage when compared to on premise uptime is when the downtime occurs. Its far more impactful if its in the middle of the working day or during the busy period of shopping. A store that goes offline in the middle of black friday or in the run up to Christmas is harmed a lot more than some down time on a Sunday night/Monday morning at 3am.
One thing I have noted over time is a lot of these AWS, Azure et el downtimes is they occur in the middle of everyones day, millions of people are impacted by them. Same with github its getting in the way of work. Whereas when we hosted services on our own equipment the downtime was usually out of main hours. The percentages are in many ways the wrong measure of downtime because hours aren't equal in impact to businesses.
It's fairly safe to assume that most outages occur at the worst time possible. They tend to occur at times of high load.
Surely there's a word for this, but I haven't learned it.
The problem with load balancing also shows up in employees mass quitting. It's the same queuing theory problem but I don't know the name of it.
Essentially, when you run a set of 'resources' near capacity, if one fails then the next most highly loaded one also goes past capacity and also fails. The work keeps getting foisted onto other units that also fail.
In retail, the beleaguered employee quits in frustration, and the three other people who can do the same onerous tasks the quitter did now find their jobs have gotten 33% worse with no extra remuneration for the insult, and one of them quits too. The last one or two simply refuse to do the task more than they used to, and the boss can't threaten them with being fired or reported for it because if they quit then the business ends. And still the whole thing goes off the rails.
That's why consistent hashing divides the work of one failed node and spreads it evenly over the remaining nodes, so that a spike is softened as much as possible instead of being concentrated on one or two fallbacks who also immediately fail. It's not that it's a great solution, it's that it's the least bad option amongst some pretty terrible ones.
The solution is to undersubscribe the hardware, which some beancounters hate. But most bean counters don't understand Queuing Theory.
Its not for you, its advertisement, to preemptively avoid both consumer and securities fraud accusations, and for their enterprise client’s IT security review and SLAs
Mentally convert to downtime: 99.8 is clearly twice as bad as 99.9
Similar for LLM measures from an ideal 1.0 mark.
These service status pages have so many other problems, I can't really get excited about this article's point. As a user of a service, I don't really care that an additional 0.09% of uptime is any more or less difficult to achieve than an additional 0.9%, even if you describe it in terms of fractions of time. I only care about two things: what the service status is right now and your service reliability's impact to me over the long term (get out of here with your 30-day crap).
In the electric utility world we have a few IEEE standardized metrics (with appropriately IEEE'd acronyms) for tracking service reliability that I like much better and always wish for when I'm looking at a status page. Pie in the sky stuff for sure, nobody wants to do this analysis and publish the results without a regulator telling them have to, but c'est la vie.
SAIDI - System Average Interruption Duration Index. How many minutes an average customer experienced service interruption in a year. This is the big one I'd want to see on your service status page IMHO. For the power grid, we consider any outage longer than five minutes to be an interruption ("non-momentary outage").
SAIFI - System Average Interruption Frequency Index. How many total periods of interruption occurred for the average customer in a year.
CAIDI - Customer Average Interruption Duration Index. How long it takes service to be restored for the average customer when there is an interruption.
For the US, here is what these numbers look like: https://www.eia.gov/electricity/annual/html/epa_11_03.html. If you're outside the US look up yours and have a good laugh at us. :)
For the "right now" aspect you have probably visited your utility's outage map, but here I would say we do much better than most utilities. The level of detail on the investigation and resolution is often more detailed, and we usually know better than to bother providing much in the way of a concrete estimate for restoration time of a current outage (though this is getting better in the utility space).
If you want to feel better about that, check out South Africa. You know you have a problem when "load shedding" is a household word.
(I should note that things have improved recently: it was a big milestone when they went a full year without load shedding, as of May 16.)
We've stopped with our caring about quality and understanding so I don't see why we'd keep caring about reliability and availability. How are you going to pinpoint performance issues when you have no idea how your app works?
Point the AI at your logs, tell it to fix things, rinse and repeat. It's faster than debugging, and fast is good.
Fast is future. Slow is ded. We are fast now
I think this idea is objectively good, but practically hazardous. "Four nines" is a meme at this point, and it's a helpful one: a commonly understood gold standard. Few people know that translates to roughly 4 minutes and 19 seconds of downtime per month. Since service quality is degrading, I fear that changing the way we express those uptime might also unintentionally provide convenient cover to reset expectations, e.g. 10 minutes of downtime is the new norm for reliability. Or 30 minutes. I don't think we should yield the Overton Window, so to speak, to this.
But if few people know that 99.9999 == 4m19s, then how is anyone losing out on changes? If you hadn't told me and I didn't have a need to calculate it with certainty, I'd have believed you if you'd said it was an hour a month.
I think the point is that the metric being meaningless right now to most people makes it easier to shift the reliability-Overton-window. 99.9999 vs 98.0 seems not to bad to a layman (I.e. executives), but 4m19s to 12 hours seems pretty intuitively bad. Sure, we may lose some shift in the immediate future, but it's easier for that to continue happening with just percentages, is the point.
It's not like we have to stop showing percentages as well, but time is a much more accessible expression. Right now the lay-ness of company leadership has already allowed the shift to happen pretty markedly.
I’ve worked in systems which at worse have had three nines for years, but I’ve also worked in systems where five nines is a failure.
This attitude of modern tech claiming 98% is good just doesn’t work in the old tech acceptance. We had individual components fail all the time. We’re still looking at a 230ms outage to a branch office last week caused by a power failure combined with a badly plumbed power distribution.
Modern software people don’t consider 230ms to be an outage. Glad they don’t work in electricity.
(The failure we had was only on the services we guarentee at 99.1%, our lowest sla. After that there’s 99.95 and 99.999.
(In reality we reach five nines year after year on even the lowest levels, but there are major concerns like “large bomb in data centre” which could cause some of our less critical units to drop way more than 5 minutes a year.
That's not really an old versus new thing. The water company isn't going to consider 230ms an outage either.
With tech, like, it's crazily difficult to make sure every http request succeeds, so you build in retry, and look at that 230ms doesn't cause real disruption now. Not to excuse 98%, that's awful.
They want to move goal posts of integrity because it makes their slop factory value proposition work better
Funny this was posted a few hours after a Salesforce global outage
I wrote a blog [1] last week about what we've been doing at Depot to work around GitHub's instability. I wrestled with how to present their downtime in a way that communicates its real impact to users, and I landed on a similar approach as the author. But I think even this doesn't state it strongly enough. Their uptime stat comes from their public status page, so it's conservative by definition. And it's important to remember that their downtime comes almost exclusively during business hours, when people are relying on the platform to get work done.
There are projects like The Missing GitHub Status Page [2] that attempt a more accurate reporting, but taking 98.31% for granted, I think the more impactful framing would be _a floor_ of 1.5 business days lost every month. At 22 business days per month, that can feel more like 7+% downtime.
[1] https://depot.dev/blog/github-actions-without-github
[2] https://mrshu.github.io/github-statuses/
You should be slightly careful about that.
I saw a User's Group presentation by a guy at Speakeasy, they had made a whole logistic system that looked a lot like a CRM manager but not for customers, for dealing with how flaky Covad was at the time. Covad only had to be less shitty than Qwest (who we called, "Qworst", because if you hadn't dealt with Covad, they were the worst) and they couldn't even manage that.
So if Covad promised to have some line work done by 10 am on Tuesday, the SE techs would get a reminder to verify in had been completed shortly after the deadline so they could call Covad before the customer noticed the install was behind schedule and ask why it hadn't been done by the deadline. Otherwise SE was telling people they'd have Internet in 4 business days and it would be more like 10 - on average.
Around the time Qworst became CenturyLink, Covad bought SpeakEasy, for a net loss of beauty in the world.
Not sure if some one already mentioned, these numbers are rarely for general public
1. These are core to the (service level agreements) SLA's for the enterprise companies when they evaluate and sign the contract (must for government RFPs)
2. The contracts generally have provisions for payback or penalties for missed SLAs
3. When your product depends on any of the products directly then the availability of your product has to take into account the availability of the dependent services. Eg, I can easily sell a SLA of two nines for my service if I am building it on a platform which has SLA of three nines, other way around is always questionable (though possible)
4. All of this is what used to happen may be up until 5 years ago. As of today even with good number of outages, company like AWS has not updated their availability to reflect those outages. Ideally all the companies built on top of AWS infrastructure would have to update their availabilities in a cascading manner but some how everyone decided to skip the beat
5. At this point, calculation of the availability itself has become so opaque that no one understands it anymore so no one questions it either
Many commenters seem to think the author is saying that we should stop caring about uptime, when what he's really saying is that we should be reporting time (i.e. number of hours down in the last month) rather than just reporting an opaque percentage that hides the problem.
I want a good chart not a percentage, because one service being down at 2 am for 3 hours is not the same as another service being down for 45 minutes during my peak hours of productivity.
One advantage of percentages is that you can multiply them if the outages aren't related. If you rely on 10 systems with a 99% uptime each you end up with roughly 90% uptime in total.
After a merger we ended up with 2 operations systems and one of them thought the other one were clowns. But they thought everyone were clowns so it was hard for the rest of us to tell. They never actually apologized for being assholes, but their tune changed when they found they had bitten off more than they could chew later and they had to come hat in hand to a bunch of groups to delegate responsibilities back to them.
So a production outage happens, and we go to look at our runbooks to figure out what to do. The Wiki is also down, so no runbooks. What the actual fuck?
So it turns out the alleged clowns put all of our internal infrastructure onto the same SAN system in different partitions. They got a lot less judgy and the other Ops team got a lot more autonomy after that event.
Anyone with an Operational IQ above room temperature knows that you don't put offline and online resources onto the same hardware. Not only do they have separate duty cycles but also offline services can end up taking out the online services in unexpected ways, in part because they are assumed to be a bit sloppy and so they get less operational scrutiny.
I sped up the average runtime of a batch job by quite a humongous amount by throttling the request rate we made to a service that was used by user-facing services. If several unrelated batch jobs kicked off during the wrong time of day, we would start getting circuit breakers opening everywhere, including occasionally production.
I throttled us to use something like 8 or 10% of the official capacity of the service and no more. But I did it as limiting the number of in-flight requests, so that created back pressure if the servers were under heavy load and sending responses slowly, or let us go faster if the servers were returning results faster than usual. That peak shaving saved the other team a load of grief and let them push some capacity planning work down the roadmap to work on other things. And it dropped our failure rate by more than a factor of six. Low enough that we no longer had to babysit that process, which meant we started using it much, much more.
100%!
Reporting absolute instead of percentage values definitely makes it more human. But most of these reports & status pages are held up-to-date manually, by engineers, when incidents happen. I’d bet not even half of the real outages end up reported on there.
“It’s just a few minutes of downtime, nobody noticed, best not draw attention to ourselves by placing something on our status page”
Status pages should be tied into (external/simulated) monitoring to update in near real-time, without hiding incidents. Transparency is the only way. By reporting the _real_ hours of downtime, including the small stuff you think people didn’t notice.