Hacker News

New stories

Live mirror
30 storiesupdated just nowView source snapshot
  1. Build and deploy long-horizon agent like Devin in 10-mins(cayu.dev ↗)
    discuss
  2. Air-FI: Generating Covert Wi-Fi Signals from Air-Gapped Computers (2020)(arxiv.org ↗)
    discuss
  3. NanoJev(github.com/tianyucodings ↗)
    discuss
  4. VitePress(vitepress.dev ↗)
    discuss
  5. Uber ordered to pay $40M for woman killed after driver left her on highway(bbc.com ↗)
    1comments
  6. Game UI Database(gameuidatabase.com ↗)
    discuss
  7. Cognitive resilience helps to predict Alzheimer's dementia(nature.com ↗)
    discuss
  8. I Hijacked a Real Artist's Spotify with AI Music. It Was Disturbingly Easy(404media.co ↗)
    discuss
  9. Non-Obvious Nix Options(matthewbrunelle.com ↗)
    discuss
  10. Show HN: Timeline, a team project planner with a fast canvas Gantt view(github.com/vakahnke ↗)
    discuss
  11. The Last Lecture (Randy Pausch) [video](youtube.com ↗)
    discuss
  12. Open-weight models take 56% of token volume, Astra doubles Fable 5.1 spend(vercel.com ↗)
    discuss
  13. Meta glasses captured and shared intimate images without consent, lawsuit claims(latimes.com ↗)
    discuss
  14. Tiny-Vedas: a generic accelerator interface for RISC-V, from PyTorch op to GDS(github.com/spzbrnmrc ↗)
    discuss
  15. Adam Tooze on AI Regulation [video](youtube.com ↗)
    discuss
  16. Sea lion found beheaded at San Francisco's Ocean Beach, prompting investigation(abc7news.com ↗)
    discuss
  17. Time To Go – an alarm that shows how long your commute takes(apps.apple.com ↗)
    discuss
  18. Cackle Pop Roar Reptiles have a lot to say(knowablemagazine.org ↗)
    discuss
  19. GLM-5.3-FlashX: Delivering inference speeds of 200 tokens/s(z.ai ↗)
    discuss
  20. Reinventing issue tracking: Local-first and Git-native(manganin.dev ↗)
    discuss
  21. There's no point at which turning your brain off will work(danluu.com ↗)
    6comments
  22. Palantir, "Effective Software" and the Quest to Make Progress Programmable(arenamag.com ↗)
    1comments
  23. Chinese AI Radicalizes(chinatalk.media ↗)
    discuss
  24. The Download: AI's extinction risk and bioweapons threat(technologyreview.com ↗)
    discuss
  25. A Pattern for Portable Agent Configuration(cameronboehmer.com ↗)
    discuss
  26. Parents are interfering with teens' first jobs. Employers say it can backfire(cnbc.com ↗)
    discuss
  27. Tickrs: Realtime stock ticker data in your terminal(github.com/tarkah ↗)
    discuss
  28. Code of Artificial Intelligence Offenses(syntheticjudiciary.com ↗)
    discuss
  29. Bringing Correctly Rounded Math to Production with LLVM-Libc(devblogs.microsoft.com/cppblog ↗)
    discuss
  30. Photon-Emission-Guided Laser Fault Injection Enables RP2350 Secure Debug(ledger.com ↗)
    2comments

NATS publishes preliminary report on technical incident of 8 September

32 pointsby 4h agonats.aero
23 comments
2h agoHN ↗

Note: NATS the UK airspace manager, not NATS the pubsub system.

1h agoHN ↗

And here I was surprised they would use NATS in their implementation to assign squawk codes.

2h agoHN ↗

The incident was triggered by a valid manual request for a squawk code. This manual request was made correctly and there was nothing abnormal or invalid about the associated flight plan.

While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted.

The reason this scenario has not occurred before is because:

1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond.

2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value.

3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.

Post-incident investigation has identified that when processing of the squawk allocation request resumed, the data associated with it had been corrupted and affected some subsequent flight data updates.

1h agoHN ↗

3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.

Well, that's comforting to know.

BBC: "Flight chaos caused by software defect in space of a millisecond, report says"

Sky: "'Millisecond' software error caused air traffic outage that grounded thousands of flights"

The Guardian: "Flight chaos for hundreds of thousands was caused in ‘millisecond’ by software error"

Sounds like pure bad luck.

1h agoHN ↗

Testing isn't an effective way to catch most race conditions. Code reviews, static analysis, and rigorous enforcement of concurrent coding standards is usually a better approach.

1h agoHN ↗

Perhaps something like what TigerBeetle does: deterministic simulation testing.

1h agoHN ↗

Or maybe even just reviewing logic that is subject to pre-emption.

Maybe I'm being too harsh.. on the plus side the system has at least failed hard every time there's been a fault. Nobody has died. But it's been 3 times now in the past couple of years, and two of those times resulted in over 2000 flights cancelled and days of backlog, and misery for hundreds of thousands. It's really not acceptable.

1h agoHN ↗

the system has at least failed hard every time there's been a fault

You don't know that.

1h agoHN ↗

Safety critical systems demand formal verification. This wasn't bad luck, this was poor craft.

1h agoHN ↗

Therac-25 called and wants its bug back.

Seriously, for 2026 this is pure amateur hour with no excuse.

34m agoHN ↗

2026 … but the code is running from what decade and has to interact with systems from what other decades?

54m agoHN ↗

Very interesting. I spent a large part of my career in aerospace and never considered this failure mode before. It makes me wonder: how long is the pathological code allocation time? A few seconds, at most? We're talking about flight identification codes that are normally assigned upon takeoff and change at most a handful of times during a flight.

I assume the "manual request" is an aircraft squawking 7700 or similar, but why does the system need to interrupt an in-flight allocation in the first place? Any controllers here have insight?

One would think it would be sufficient to do something single threaded like

    if(!highPriorityQueue.empty() {
        highPriortyQueue.processOne();
    } else if(!lowPriorityQueue.empty()) {
        lowPriorityQueue.processOne();
    }

or whatever, but they're not and I'm curious why.

31m agoHN ↗

there will be some other tasks in the system that need time guarantees around when they get scheduled for very important (safety?) reasons. so when these higher priority tasks show up the lower priority tasks get suspended.

if they could guarantee hard bounds on how long low priority tasks take to complete they could implement your scheduling algorithm. but i think in reality the low priority tasks are either not boundable or they if they do have a provable bound the bound is too high.

26m agoHN ↗

It’s my understanding that this subsystem that failed only allocates codes but I could be wrong there.

It doesn’t make sense to have other non-code allocation things competing for queue space with code allocations.

40m agoHN ↗

3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.

How often does that original request happen per day? How often does the higher priority activiry take?

If the original request happens 864 times a day and the high priority request ten times, there's a 1 in 25 chance it will happen in a given year.

2h agoHN ↗

Interesting! Looks like a race condition (with a 1ms window) during squawk allocation (the process that gives each aircraft a 4 digit identifier for ATC purposes) caused data corruption. They present that window as being small but 1ms is a decent chunk of time in our work lol

1h agoHN ↗

More specifically: it sounds like something which was supposed to be an atomic operation was split into two by the preempting task, with corruption happening because that task also interacted with the same piece of (now-temporarily-invalid) data.

1h agoHN ↗

Would be interesting if the report actually got into the details. The curiously specific "approx. 1ms window" sounds like an off-the-shelf memory race condition in combination with a preempting thread, for which 1ms is an absolute eternity and Guaranteed To Happen By Tuesday This Week™. But the "database corruption" sounds like database IDs being generated from timestamps with 1ms granularity.

1h agoHN ↗

They might have to switch to ULID or similar.

34m agoHN ↗

A few years back, NATS went down because a flight plan waypoint confused the system and it crashed. They had to manually find the problematic flight plan, remove it and start the whole system back up.

Considering the last issue they encountered, it looks like in more than one place, there is no error catching and graceful resolution for those errors.

I would assume a system of such importance to handle issues without hiccups and alert the operators of what did not work. Like “this input caused this problem”, not just crash.

14m agoHN ↗

Back in 2023 when the previous issue happened, it didn't actually "crash", it detected what it perceived as an inconsistency (which was due to invalid waypoint logic for waypoint codes in multiple countries) and put itself in "maintenance mode".