I don't really care about percentages, either. But for some industries, the difference between "two 9s" and "five 9s" can be millions of dollars, so that's why they're published that way to the customer.
What's the point here? That everyone should use the n-nines notation? Sure. However, companies have no interest in doing anything that makes them look worse.
Also, is anyone else getting the bitter taste of AI writing from this page?
Services can have a 50% uptime (or a 50% downtime if you prefer) as long as it's the time when I need it to be up (or don't need it.)
Which is to say that the significance of downtime depends on the user. Talking about nines only makes sense internally when you are evaluating your infrastructure and operations. It doesn't tell you squat about impact to your customer.
Right. And there's nothing wrong with having regular scheduled downtime. That's obviously not appropriate for most massive scale global cloud services, but it's great for services that know their usage patterns well.
GitHub Actions: 12 hours affected in the last 30 days (98.31% uptime).
This is trying to shine the most favorable possible light onto a deteriorating situation. It doesn't take away from the fact that most businesses have measurable missed revenue in downtime. Customers that shop somewhere else, ads that were never severed, leads that grew a little colder. 12 hours of downed GitHub results in millions of dollars of lost developer productivity that was externalized by Microsoft to other companies.
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
One thing I would like to see is how many of those hours are during normal business hours in my country.
Not all hours are created equal when it comes to downtime and my intuition is that most of these 12 landed within my working hours.
In terms of impact that then might mean they were down for 7.5% of the time I needed them, or had business hours uptime of 92.5% which is… both not very good and very disruptive.
On the other hand, downtime at 4AM would be much less impactful even if it happened every day and added up to more overall downtime.
Then again if you’re a business relying on GitHub enterprise you have an SLA and you can and WILL charge GitHub for failing their SLA. Usually there’s a real measurable dollar value tied to that SLA per dollar and it’s not cheap.
What surprises me in particular is that the global API and the GitHub EE API are the same which is a big no no. This is even more surprising given the fact that paying GitHub customers are clearly the minority both in numbers and code velocity.
My assumption would be that GitHub is keeping EE up and the rest of free or pro users just have to suck it up. If that’s not even the case then it’s only a matter of very short time until GitHub will see businesses leave to more reliable competitors
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
They're also completely irrelevant, you as a customer of a service that is down can lose the same amount of money in a 5 minutes outage or 30 days outage, if you were only relying on this service for one operation that took 1 second and had to happen during the time where the outage happened.
Depending on the service in question, no amount of downtime is acceptable, however unrealistic this is.
For example we had a 6 9 (99.9999%) requirement from a customer for any given 3-6 month period. If we violated that, we owed them their money back (baring the outage wasn’t caused by us - I.e our cloud provider shit the bed).
That’s something like 7.5 seconds. For a contract over $1.5M. Am I the only one who thinks that’s outrageous expectations?
I worked in realtime trading. No. Not at all outrageous. Quite reasonable actually. If that’s what we agreed and I need you to be reliable I will charge you back for being unreliable. I’m happy to pay top and extra dollar for the SLA but that means it needs to be acted on.
Well, someone on the business side of the house is free to negotiate. Whether engineering learns about the contract before sales has inked a 6-nines availability guarantee varies wildly by the company
The job of the uptime numbers are to look good (and sometimes to meet contractual obligations), more context doesn't make them sound better. Not being understood in layman's terms is a feature.
These companies are happy that you don't know the difference between 99%, 99.9%, and 99.99% and that you think they all sound pretty good.
As more and more things we might consider "platform" move to the cloud, I think it also matters what the service provider means by saying it's up. Just because the servers are alive and responding doesn't mean the platform is really functional.
One vendor in particular we deal with has a powerful feature which we use to a large extent. Unfortunately, that particular feature is all too often not working. The servers are up and the rest of the platform is working, but we need that feature, so if it's down, it doesn't help much that the rest of the platform is up.
These numbers are useful proxies for how likely you are to have your work disrupted outside of your own control.
If you do something 100 times a day against a four-nines service, you can reasonably expect that everything will succeed.
If you do something 10,000 times a day against a two-nines service, you can expect to hit a substantial number of errors during that day, or even have long periods where your work cannot happen at all.
People aren't frustrated with Github because Github has 98% uptime or whatever the specific number is. They're frustrated because it regularly interferes with their ability to work. The 98% number is just a concise way to say it.
Separately from how you present the number, the very concept of "uptime" as a single number is a bit muddy in the context of a distributed system, where different components can be differently available for different users.
Also, 0.1% downtime in the form of a 45-minute outage per month is very different from 0.1% of requests failing in brief bursts. You often see downtime reported as "increased error rates" which is so vague as to be meaningless.
So how about, and I’ll just throw this out there, instead of:
>
> GitHub Actions: 98.31% uptime.
>
> We say something like:
>
> GitHub Actions: 12 hours affected in the last 30 days (98.31% uptime).
The suggested format is equally unhelpful.
You can get 12 hours of downtime by being down once for 12 hours, or 144 times for 5 minutes. The user experience is VERY different in those two cases.
I don't really care about percentages, either. But for some industries, the difference between "two 9s" and "five 9s" can be millions of dollars, so that's why they're published that way to the customer.
What's the point here? That everyone should use the n-nines notation? Sure. However, companies have no interest in doing anything that makes them look worse.
Also, is anyone else getting the bitter taste of AI writing from this page?
Nah. Jim is just a decent writer (and historically has been fairly suspicious of AI)
Skip to the last 2 paragraphs
Services can have a 50% uptime (or a 50% downtime if you prefer) as long as it's the time when I need it to be up (or don't need it.)
Which is to say that the significance of downtime depends on the user. Talking about nines only makes sense internally when you are evaluating your infrastructure and operations. It doesn't tell you squat about impact to your customer.
Right. And there's nothing wrong with having regular scheduled downtime. That's obviously not appropriate for most massive scale global cloud services, but it's great for services that know their usage patterns well.
Can we post uptime stats instead? I just axed two of my servers from colocation two days ago. Sad to see them go, six years of FreeBSD.
No since kernel live patching is not universal it promotes bad security practices to maximize the uptime of a single server.
This is trying to shine the most favorable possible light onto a deteriorating situation. It doesn't take away from the fact that most businesses have measurable missed revenue in downtime. Customers that shop somewhere else, ads that were never severed, leads that grew a little colder. 12 hours of downed GitHub results in millions of dollars of lost developer productivity that was externalized by Microsoft to other companies.
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
I don’t see why you think “12 hours affected in the last 30 days (98.31% uptime)”
Is trying to spin anything. It’s making easier to see the impact over the last 30 days. I agree with the article
One thing I would like to see is how many of those hours are during normal business hours in my country.
Not all hours are created equal when it comes to downtime and my intuition is that most of these 12 landed within my working hours.
In terms of impact that then might mean they were down for 7.5% of the time I needed them, or had business hours uptime of 92.5% which is… both not very good and very disruptive.
On the other hand, downtime at 4AM would be much less impactful even if it happened every day and added up to more overall downtime.
Then again if you’re a business relying on GitHub enterprise you have an SLA and you can and WILL charge GitHub for failing their SLA. Usually there’s a real measurable dollar value tied to that SLA per dollar and it’s not cheap. What surprises me in particular is that the global API and the GitHub EE API are the same which is a big no no. This is even more surprising given the fact that paying GitHub customers are clearly the minority both in numbers and code velocity. My assumption would be that GitHub is keeping EE up and the rest of free or pro users just have to suck it up. If that’s not even the case then it’s only a matter of very short time until GitHub will see businesses leave to more reliable competitors
Favorable spin? The article's point is that "12 hours a month" makes the cost very obvious, and I agree with the article.
They're also completely irrelevant, you as a customer of a service that is down can lose the same amount of money in a 5 minutes outage or 30 days outage, if you were only relying on this service for one operation that took 1 second and had to happen during the time where the outage happened.
Depending on the service in question, no amount of downtime is acceptable, however unrealistic this is.
Yeah, this is always fun. Logarithmic graphs of downtime, people.
The 12 hours out of 30 days seems like sugarcoating the issue.
Keep the percentages, and regardless of that - GitHub fix your uptime
I'm also not sure that all downtime is really properly measured now as more and more services are connected and intertwined.
Some measure quite detailled but some just don't summarize the downtime from all providers up and below their own platforms.
`-log10(1 - uptime)`
These metrics tend to be bullshit in contracts.
For example we had a 6 9 (99.9999%) requirement from a customer for any given 3-6 month period. If we violated that, we owed them their money back (baring the outage wasn’t caused by us - I.e our cloud provider shit the bed).
That’s something like 7.5 seconds. For a contract over $1.5M. Am I the only one who thinks that’s outrageous expectations?
I worked in realtime trading. No. Not at all outrageous. Quite reasonable actually. If that’s what we agreed and I need you to be reliable I will charge you back for being unreliable. I’m happy to pay top and extra dollar for the SLA but that means it needs to be acted on.
It's a contract, you're free to negotiate?
If you agree to those terms knowing it's unrealistic, you're agreeing to give away your service for free.
Well, someone on the business side of the house is free to negotiate. Whether engineering learns about the contract before sales has inked a 6-nines availability guarantee varies wildly by the company
The job of the uptime numbers are to look good (and sometimes to meet contractual obligations), more context doesn't make them sound better. Not being understood in layman's terms is a feature.
These companies are happy that you don't know the difference between 99%, 99.9%, and 99.99% and that you think they all sound pretty good.
As more and more things we might consider "platform" move to the cloud, I think it also matters what the service provider means by saying it's up. Just because the servers are alive and responding doesn't mean the platform is really functional.
One vendor in particular we deal with has a powerful feature which we use to a large extent. Unfortunately, that particular feature is all too often not working. The servers are up and the rest of the platform is working, but we need that feature, so if it's down, it doesn't help much that the rest of the platform is up.
These numbers are useful proxies for how likely you are to have your work disrupted outside of your own control.
If you do something 100 times a day against a four-nines service, you can reasonably expect that everything will succeed.
If you do something 10,000 times a day against a two-nines service, you can expect to hit a substantial number of errors during that day, or even have long periods where your work cannot happen at all.
People aren't frustrated with Github because Github has 98% uptime or whatever the specific number is. They're frustrated because it regularly interferes with their ability to work. The 98% number is just a concise way to say it.
Until there is an industry wide definition of outage, degraded performance, etc then it's all moot
Separately from how you present the number, the very concept of "uptime" as a single number is a bit muddy in the context of a distributed system, where different components can be differently available for different users.
Also, 0.1% downtime in the form of a 45-minute outage per month is very different from 0.1% of requests failing in brief bursts. You often see downtime reported as "increased error rates" which is so vague as to be meaningless.
Google's "windowed user-uptime" attempts to deal with this a bit better, by exposing different views of the data instead of trying to condense uptime into a single number: https://www.usenix.org/system/files/nsdi20-paper-hauer.pdf
You can use -log10(1-p). On the "nines" it is exactly the number of nines you have:
(Modulo floating point issues of course.)
Which then smoothly covers the entire space:
But good luck getting that standardized.
The suggested format is equally unhelpful.
You can get 12 hours of downtime by being down once for 12 hours, or 144 times for 5 minutes. The user experience is VERY different in those two cases.
Ultimately the graphs are the most useful format.
Uptime % has the benefit that it's easy to understand.