I saw the talk version of that and it has lodged in my memory. The point he made was that "health.gov" was made by a bunch of teams that made sure that their part worked, but nobody was in charge of making the whole thing work.
This is sadly "normal" for large bureaucracies.
Every time I see a giant, multi-million dollar catastrophe, it's always "proper", "documented", "enterprise", "by the book", and... "a total failure". The reason is always that nobody actually cares about the final outcome, only the paperwork in front of them that they need fill out, the checkbox that needs to be ticked, or the compliance requirement that needs to be met.
This, in my opinion, is some part of how we are schooling people - do as you are told and follow the rules and some part of not having stakes in the outcome.
There's a brief period when you join any company where you really want to improve things, and then about a year in the system has asserted towards the mean.
Nothing to do with schooling, every thing to do with business structures.
Definitely not on purpose! I keep forgetting that ChatGPT does that now, which is rather irritating because I don’t use it to write comments but the tracker makes it look like I do.
I do use ChatGPT to resolve vague memories of articles read long ago into concrete URLs I can share, which is one of the best uses of the accursed things right now!
I assure you that I am an organic meat human like yourself and the article dates itself to the “before times” and can also be trusted to be free of contamination.
They're pretty readable, I recommend that second one (though it leaves some questions I have unanswered). It's a good example of how latent issues with a large system can go unrecognised for a long time before a seeming unrelated change can trigger a cascading failure
One thing it doesn't make clear – the GPS card only supported the original L1 signal, which is what causes the GPS week rollover issue. The newer L2C signal has a much longer rollover period (157 years vs 19.6 years), which means no rollover until next century. If the GPS card had supported the newer L2C signal, then likely this would not have happened even if the other misconfigurations had still occurred.
Is it really so that Time Division Multiplex timing in cellular networks is based on NTP time? I stopped reading after the article implied this - it doesn't sound plausible at all.
I would assume it’s similar to dab etc, and it’s ptp that carried that data on IP. In the broadcast tv world we use ptp, or more traditionally black and burst, or word clocks in audio.
The source is still the same - your grandmaster (ptp) or sync generator (b+b) is fed from a good timing source - often GPS, with one or two inputs, but sometimes local atomic clocks.
I’ve seen issues before with 40ms audio gaps when a GPS antenna has died and the coders can’t cope with frame drift so they incentive a 40ms silence in the audio every 40 minutes or so as the source and destination clocks are different speeds by about 1 part in 60,000.
In DAB the tolerances are far lower, there was a problem about 10 years ago due to a GPs satellite being a few microseconds off
No. The official report seems to suggest that software in IMS core (the subsystem that controls VOIP signalling in the mobile network) bugged out after the time was shifted back 20 years. That doesn't narrow down the specific failure that stopped calls from going through very much. IMS core is full of timers, but it could also have been a license server issue for all the report said about it.
I haven’t read the report, but I did read at the time that it was a certificate validation failure: clients thought the current time was before the "Not Valid Before" date. I forget now what essential service became unavailable because it was disbelieved.
(Source would either have been the Whirlpool forums – where Australian ISP and telco engineers can be found – or local Australian mass media.)
Human made summary: A telecom company had time synchronisation issues in a location of their infrastructure. They eventually used a single hardware GPS clock that fixed the issues but became their only authoritative clock in this location.
One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.
Apparently a GPS clock can keep track of the time if it’s up and running when the week counter overflows, otherwise it has to take a wild guess. Apparently their hardware GPS clock used a hardcoded start time from its firmware instead of trusting a less reliable existing clock.
The article finishes with some AI looking suggestions to prevent such an issue to happen again.
Mine would be to have bought one or two more GPS clocks and not from the same provider.
One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.
As I pointed out in my other comment below, this is only true of the old L1 signal, not the newer L2C signal. If they had a newer GPS card, this would never have happened.
Mine would be to have bought one or two more GPS clocks and not from the same provider.
I think it would be more important to have a newer one that doesn't have this problem, than two old ones which both do.
I don't think every GPS receiver necessarily has to have this problem, even with the old signal. It's not crazy to keep a week count in persistent memory and use that as a lower bound on startup so you only get a wrong time if the receiver isn't online for 1024 weeks.
Exactly. The sane ones just ask you what year it is at startup, and use that to determine the initial epoch. Then they handle rollover automatically since then.
It's been widely opined that 1024 weeks (~20 years) is the worst possible interval. Either rollovers should've happened VERY frequently (say, 128 weeks) so receivers would be FORCED to deal with it, or extremely infrequently (16384 weeks?), so it's simply never an issue.
The unhappy medium is long enough that developers feel justified in saying "naaaah, our receiver won't still be in use then, we can ignore that!", but in practice it's very likely to happen.
Years ago while working as a contract sysadmin for a clearing firm, a coworker made an overbroad puppet change that changed every server's time zone from UTC to CDT. The only thing I heard about it was from a text from an old colleague/friend who was troubleshooting why their clearing reports were all off by five hours. I still chuckle at that random text.
Shout out to the Jeff Geerling video about this
https://www.youtube.com/watch?v=1T9xQy-dsQo
Author needs to decide which end of the stratum stack they want as the top.
This is the perfect description of Telstra as a company.
This reminds me of this: https://obamawhitehouse.archives.gov/blog/2015/03/26/why-we-...
I saw the talk version of that and it has lodged in my memory. The point he made was that "health.gov" was made by a bunch of teams that made sure that their part worked, but nobody was in charge of making the whole thing work.
This is sadly "normal" for large bureaucracies.
Every time I see a giant, multi-million dollar catastrophe, it's always "proper", "documented", "enterprise", "by the book", and... "a total failure". The reason is always that nobody actually cares about the final outcome, only the paperwork in front of them that they need fill out, the checkbox that needs to be ticked, or the compliance requirement that needs to be met.
This, in my opinion, is some part of how we are schooling people - do as you are told and follow the rules and some part of not having stakes in the outcome.
No this is what the corporate world does to you.
There's a brief period when you join any company where you really want to improve things, and then about a year in the system has asserted towards the mean.
Nothing to do with schooling, every thing to do with business structures.
This was inspiring, thanks for sharing.
Why did you include the source tracker in your link?
Definitely not on purpose! I keep forgetting that ChatGPT does that now, which is rather irritating because I don’t use it to write comments but the tracker makes it look like I do.
I do use ChatGPT to resolve vague memories of articles read long ago into concrete URLs I can share, which is one of the best uses of the accursed things right now!
I assure you that I am an organic meat human like yourself and the article dates itself to the “before times” and can also be trusted to be free of contamination.
ChatGPT is a far more useful search engine than Google, duckduckgo, etc.
It won’t last, but just as Google was so much better than hotbit, altavista, yahoo etc back in the day.
This is the perfect description of most legacy telecoms everywhere.
Lots of bloviating AI-prose. Here's the original report:
https://www.telstra.com.au/exchange/what-we-ve-learned-from-...
https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr...
They're pretty readable, I recommend that second one (though it leaves some questions I have unanswered). It's a good example of how latent issues with a large system can go unrecognised for a long time before a seeming unrelated change can trigger a cascading failure
One thing it doesn't make clear – the GPS card only supported the original L1 signal, which is what causes the GPS week rollover issue. The newer L2C signal has a much longer rollover period (157 years vs 19.6 years), which means no rollover until next century. If the GPS card had supported the newer L2C signal, then likely this would not have happened even if the other misconfigurations had still occurred.
Direct link to the analysis report: https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr...
All I could think of while reading this was “has no one heard of ntptrace?”
It's funny, but I had the same thing happen to me once. One day, overnight, my computer decided it was 2006.
It happened twenty years ago...
;-)
Is it really so that Time Division Multiplex timing in cellular networks is based on NTP time? I stopped reading after the article implied this - it doesn't sound plausible at all.
No, it absolutely is not.
That timing requires a very tight time, frequency and phase reference. It’s derived from local high stability GNSS-disciplined time references.
I would assume it’s similar to dab etc, and it’s ptp that carried that data on IP. In the broadcast tv world we use ptp, or more traditionally black and burst, or word clocks in audio.
The source is still the same - your grandmaster (ptp) or sync generator (b+b) is fed from a good timing source - often GPS, with one or two inputs, but sometimes local atomic clocks.
I’ve seen issues before with 40ms audio gaps when a GPS antenna has died and the coders can’t cope with frame drift so they incentive a 40ms silence in the audio every 40 minutes or so as the source and destination clocks are different speeds by about 1 part in 60,000.
In DAB the tolerances are far lower, there was a problem about 10 years ago due to a GPs satellite being a few microseconds off
https://www.radioworld.com/trends-1/gps-anomalies-caused-pro...
No. The official report seems to suggest that software in IMS core (the subsystem that controls VOIP signalling in the mobile network) bugged out after the time was shifted back 20 years. That doesn't narrow down the specific failure that stopped calls from going through very much. IMS core is full of timers, but it could also have been a license server issue for all the report said about it.
I haven’t read the report, but I did read at the time that it was a certificate validation failure: clients thought the current time was before the "Not Valid Before" date. I forget now what essential service became unavailable because it was disbelieved.
(Source would either have been the Whirlpool forums – where Australian ISP and telco engineers can be found – or local Australian mass media.)
That's also plausible.
ai;dr
Human made summary: A telecom company had time synchronisation issues in a location of their infrastructure. They eventually used a single hardware GPS clock that fixed the issues but became their only authoritative clock in this location.
One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.
Apparently a GPS clock can keep track of the time if it’s up and running when the week counter overflows, otherwise it has to take a wild guess. Apparently their hardware GPS clock used a hardcoded start time from its firmware instead of trusting a less reliable existing clock.
The article finishes with some AI looking suggestions to prevent such an issue to happen again.
Mine would be to have bought one or two more GPS clocks and not from the same provider.
This is the opposite of a Google AI as summary in the best way.
Thank you for slogging through the slop to bring the human angle
As I pointed out in my other comment below, this is only true of the old L1 signal, not the newer L2C signal. If they had a newer GPS card, this would never have happened.
I think it would be more important to have a newer one that doesn't have this problem, than two old ones which both do.
I don't think every GPS receiver necessarily has to have this problem, even with the old signal. It's not crazy to keep a week count in persistent memory and use that as a lower bound on startup so you only get a wrong time if the receiver isn't online for 1024 weeks.
Exactly. The sane ones just ask you what year it is at startup, and use that to determine the initial epoch. Then they handle rollover automatically since then.
It's been widely opined that 1024 weeks (~20 years) is the worst possible interval. Either rollovers should've happened VERY frequently (say, 128 weeks) so receivers would be FORCED to deal with it, or extremely infrequently (16384 weeks?), so it's simply never an issue.
The unhappy medium is long enough that developers feel justified in saying "naaaah, our receiver won't still be in use then, we can ignore that!", but in practice it's very likely to happen.
Coincidentally just bought myself a stratum 1 ntp/gps "clock" few weeks ago for £60.
Runs in my loft, what a time to be alive.
https://lectronz.com/products/gps-ntp-server
Years ago while working as a contract sysadmin for a clearing firm, a coworker made an overbroad puppet change that changed every server's time zone from UTC to CDT. The only thing I heard about it was from a text from an old colleague/friend who was troubleshooting why their clearing reports were all off by five hours. I still chuckle at that random text.