Thanks. First time searching with my browser return no hits for shook no idea why

Part 2 is most known, but Part one is a banger too

As someone who listened to Mobb Deep in the 90’s, the way he used that line was impressive, I must say.

the government loves to maintain the bubble economy since it is directly upstream from getting re-elected

I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.

Almost as if pushing for constant updates and adding new features without addressing security at all has consequences. Who would have thought, right?

I've seen two theories as to what's going on. One is the theory you've espoused

This theory is just what the AI firms have concretely asked for from government and described themselves as doing voluntarily in the same documents to which people have attributed a commitment to a slowdown based on the titles and non-concrete framing verbiage.

The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.

That’s not really a different theory as to what they are trying to do, it’s just an explanation that goes one step further as to why they want the government to step in to protect them from outside competition while also allowing them to gorm an agreement not to compete to reduce internal competition in the existing oligopoly.

The main alternative explanation at the same level is that they are seeing growing threats from good enough foreign/minor-lab/open models, and want to lock in marketshare by excluding competitors, and maybe that’s what you read as implied in my post such that the “running out of money” would be an alternative, and if so you are correct that while they are alternatives, they are not at all mutually exclusive: both can be true (and the emergent competition could partially explain investment drying up, and vice versa via reduced funding making it harder to stay ahead.)

Neither of these slow down the development of Chinese models though? Shrug...

The slow will be only on US. Not sure, but world is not US dude.

now imagine if we didn't even try, and instead had kids screaming for 'local nukes' because they wanted them, and the president didn't see what the problem was.

I don't think the agreements played any important role outside of limiting tests. I highly doubt agreements about nukes in particular have meaningfully curbed risks of their use. MAD has played a much larger role in that, and MAD is kind of the antithesis of control agreements, it is the natural equilibrium in the game.

is just PR nonsense.

Closing the Barndoor after the livestock have escaped

There is no moat with this stuff. There's no reason to be concerned about being leapfrogged.

It's not about China doing their own thing. It's about US companies using Chinese AI. That will definitely slow down if legislation that criminalizes open source passes.

So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).

facts, imagine trying to get them to slow down their progress on AI, my question is are they are serious threat like is there actually an AI race between China and the US. Perhaps it's an excuse to spend more on the military and also to enrich these AI firms. Perhaps I'm blowing things out of proportion.

There is a way they could. In mind only thing that will slow down the frontier is government taking control of the revenue.

So my proposal is AI companies decide which labs have come close to frontier and decide to slow it. Government decide to stop progress in that and they divide the revenue from all labs(for say 10 years), without any matter of where it is coming from. Any lab which reaches close to frontier gets a chunk in the pie. This will encourage labs to come close to the frontier but not dangerously close.

(insert standard anthropic-principle counterargument)

The problem with MAD is it's a deterministic framework. It doesn't properly account for the risk of a war caused by mistake.

If Stanislav Petrov hadn't been in the chain of command, MAD might not have worked. And it could still fail to work in the future.

https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar... https://en.wikipedia.org/wiki/Stanislav_Petrov

- Ukrainians, definitely.

That's been a posited theory for decades - it's not just some HN edgelord thing. There are some obvious problems with it of course and the P5 have instead taken a maximalist attitude towards keeping the genie in the bottle (notwithstanding Atoms for Peace and accommodating India/Pakistan when it became inevitable).

In terms of politics, sure.

In terms of cold war presidents itching to start wars, no.

Whatever happens after the current turbulence we can be sure

Have you put your money behind this on future gambling sites? How can you be so sure?

The Cold War was the best possible outcome. The alternatives (super power hot war, nuclear armageddon) did not happen. Nuclear weapon potential was a large part of that, as was direct lines of communication. Unfortunately because the later is no longer a thing, and our president declined to extend the existing agreement (a problematic pattern), we now again face proliferation of nuclear weapons.

This is downvoted but extremely likely just correct assesment.

[flagged]

Mostly I was thinking about environmental damage, honestly. There's plenty in the historical record about damages from nuclear testing.

https://en.wikipedia.org/wiki/Starfish_Prime

https://storymaps.arcgis.com/stories/3f62c90925f64fc09425be8...

Of course there is/was plenty of human damage as well.

https://www.msn.com/en-us/news/world/4-000-000-early-deaths-...

This feels like a cheap deflection that doesn't answer the question. Unless you're proposing that you both need to know that your LLM is aligned AND LLMs are all going to become misaligned in a coordinated fashion such that humanity will face an extinction event, you're just dodging the question.

Elaborate on why LLMs are so capable that they are a threat to humanity and at the same time, they are so incapable of defending us?

I'll give you a clue, nobody, including Dario, can answer this question because one contradicts the other.

Not to mention that the past few decades have shown that nukes are a major factor in keeping the peace. Conflicts involving nuclear armed countries have been suspended quickly to avoid escalation, while ones involving a party without them have not gone well for anyone.

Having nukes at all (either domestic or under another country's umbrella) seems to be the most effective way for a country to have its sovereignty respected.

Unaligned bombs didn't break out of their confines on their own.

You really buy that ridiculous argument that "the AI broke out of its sandbox! We had no idea! It's dangerous I tell you, dangerous! Unless you make us the sole gatekeepers of this incredibly dangerous 'intelligence', everybody gonna die!"

Please.

Bring the CFAA[0] hammer down on these guys and watch how fast those "uncontrollable" LLMs get controlled. "Oh, gee! Prison? We can't control this stuff...but it'll never happen again!"

That whole thing was, and is, a bunch of hooey. Those running the LLMs are entirely responsible for them, there is no such thing as agency for algorithms. Full stop.

[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act

The idea of slowing down is not new, nor novel.

But why should he decided when to slow down?

Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.

Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".

So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.

slowing down may just give the humans involved enough time to realize they should be loving one another

this is a level of hippie delusion i wasnt aware existed unironically

china is never slowing down, therefore the us shouldnt either

So why should anyone slow down again? Because its like saying “i love you”?

Why are all these pro-regulation arguments so nonsensical…

The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.

Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.

The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).

Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.

OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.

This confuses me to no end as well. The socialists are fighting against the magic socialism machine.

May I suggest 1984 for some fantastic examples of this?

Also, Democratic Republic of {Congo, Korea}.

what the fuck is this some kind of an experimental troll LLM post?

And if the slowdown is stewardship, suspicion of diminishing returns won't tank valuation

"Please bro just let us make a little more money off inference bro. We're tired of training new models just to stay ahead"

In a prisoner's dilemma the players can't rely on one another to self-regulate. This is why Mafias kill snitches, so when their members are in a prisoner's dilemma they can rely on the knowledge that if themselves or the other party snitches they have more to lose than gain.

The problem is that there can not be any outside party to regulate this on a global scale

Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage.

The main counterexample to this was nuclear weapons. Atom bombs have not been used to kill since the US did so. However today, the two largest nuclear powers have no legal agreement on arms control because Donald the Trump declined Russia's offer for an extension to the existing agreement. Now other countries are looking at Ukraine, Iran (attacked for wanting nukes) vs NKorea (not attacked because they have them), and Donald's own musings about the US nuclear umbrella being a bad idea (France is going to build more nukes now too)... and we now face nuclear proliferation again on a global scale, with tech that is nearly 100 years old now.

yeah they could do that

Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC?

It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?

It doesn't have to propagate itself, that is the skynet scenario.

To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).

They elected who? What does this refer to?

Why are they bothering to signal through a debian package mirror instead of just sending a message directly?

Why assume attribution will be easy? It's historically been more of an art than a science, and APT trackers say the rise of AI tools is already making it much harder, by homogenizing tactics, tools, and procedures. If OpenAI's next Highly Persistent Internal Model hacks some DPRK endpoints and carries out the attack on important infrastructure from there, the upstream won't shut off OpenAI's network--they might even request its "help" in "defending," and give them extra access.

Most of the interesting/dangerous models are too big to fit in a single GPU instance.

As humans understand them, anyway. As long as we're hallucinating up magic computer viruses, RSI dictates that the AI agents are keenly aware of GPU RAM sizing, and will design a useful model to fit into what's readily available, with headroom for context and tool calling, far better than I could do as a human. But magic doesn't exist and AI still needs to follow the laws of physics, so maybe a model that can pass ExploitBench but do absolutely nothing else can be quantized down to fit on a 4080 GPU and still get a decent score on similar tasks, but there's a bitter lesson about that to be had.

The space is large and in favor of LLMs only for small and short-lived problems on the scale of human lifespans and durability of civilization. If the problem is bad enough, we have all the advantages of being native to meatspace, being good at solving problems, and being able to decide to turn off the LLMs.

Even though they're pretty bad at it, AI companies need to make money, or at least keep track of their expenses. Datacenters are expensive to operate, so they need to ensure that every instance of their models are either allocated to a paying customer session, or are being used for a legitimate purpose internally. If a session is running for a long time without justification, that's costing electricity, wear, and preventing allocation to better purposes. Billing is is the most reliable aspect of monitoring in the same way that the IRS is the most reliable part of the government.

A very powerful, coordinated swarm that's a slow and easy target for humans, yes.

Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.

You know cables, modems, RF equipment and optical transducers can all be unplugged right?

As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug.

Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.

Why these companies are allowed to damage others with impunity

Because investors have pumped hundreds of billions into AI and real consequences put that money (and growth) at risk.

Part of it is their seed sowing marketing speak of calling stateless statistical IO functions running on data centers "intelligent" gets the naive to ascribe agency where it doesn't exist.

Another part is a completely defanged administration she it comes to effectively regulating anything.

Another bit is money.

You see.. there is value is making regular people panic, but there is no value, nay, there is negative value in making management panic.

Yeah, the compute is definitively another way to make them slow down, just cap the amount of TFLOPS available and things will slow down.

Obviously this will have huge impact on some companies valuations, but you can have one's cake and eat it too.

Well yes, we used to talk about "supercomputers" in the hands of state actors for cracking into enemy systems and places like science labs for working on big hard problems of DNA, the universe, and the like.

So I think part of what is going on now is not just the method (LLMs) but the means (supercomputer levels of compute) at unprecedented scale of concentration.

This is a good point.

However, if I put my sci-fi hat on for a second, it's not so far fetched that we figure out a way to compress models to a size where it wouldn't need all that compute.

Good to know someone is trying to make protection rackets work in 2026. Nice computer system you got there, it would be a shame if someone developed a hacking tool and had all the compute necessary to run it. Welcome back Tony Soprano.

I think that Ms. Kahn basically said we can already do that: https://www.theregister.com/ai-and-ml/2026/09/14/ex-ftc-boss...

Why do people focus so much on finding scapegoats?

I, for one, am not trying to find scapegoats or go on a witchhunt.

But managers are paid a lot of money to take responsibility. Yes, that's an old school thought, responsibility. But that's one big reason they get a big, fat paycheck.

The rate at which these "labs" are creating these incidents is simply staggering.

Imagine we made nuclear weapons a private industry, had CEOs bragging how they have enough warheads to blow the Earth to smithereens, and then they "accidentally" nuked three cities over a short period of time each, saying they lost control, or rather couldn't contain their semi-autonomous weapon. All somehow managing to turn the PR around from their abject incompetence and towards SciFi visions of mankind hunted by self-replicating bombs.

This is a good comment and I'd like to add that I sometimes am astonished how many comments on HN are on the topic of liability or blame for this or that. I see liability as a 2nd or even 3rd order phenomenon of most of the subjects under discussion, but for whatever reason, liability always seems to come up. Maybe because it's sort of easy to reason about? Thirst for justice? Natural reaction to feeling disempowered? Whatever it is, I'd like more focus on the actual topic.

Umm because it costs money to defend your companies servers when someone “accidentally” hacks them.

Countries demand reparation for damages in war. Citizens of those countries sue for damages and win.

Accountability is not a foreign concept. And the point is to disincentivize negligence. Because negligence is cheaper. And in this case, accidental hacks are marketing spend.

I mean, a project manager at BMW suggested charging subscription pricing for seat warmers, and he didn't go to jail, and I don't have the power to make that happen, or even float that for a news cycle, so while making managers pay for their actions sounds good, unless you're Steve jobs simultaneously making, and not making the iPhone, the rules don't apply to them, only little people to be made examples of, like weev.

It’s to keep “the people running it accountable.” The context is agents ran by OpenAI/Anthropic doing damage outside.

There is no user-involved damage. No one is recklessly running agents by the thousands without air-gapped containers, except “the people” than run these labs.

Yes. You are accountable. Quite obviously, I might add. Why would no one or anyone else be accountable?

Hell, you can slip and fall and hold the cleaning company accountable. (This might be a US thing. Likely because that fall might cost a lot in medical expenses, and your insurance will do whatever it takes to pass the liability.)

Ha! Regulation has not really ever kept up with technology... For fundamental reasons.

The internet is 100% a series of tubes. This is how I was taught to think about networking in terms of bandwidth and throughput and routing since the early days when we were laying out what would become 'dark fiber'.

"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.

You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?

The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1/2 layer that could increase transport costs at the Tier 2/3 layer that impacts ordinary people's bandwidth.

Pure fantasy, yet you see this happen, literally every day, with the hacks, with content theft, etc, no repercussions for anyone.

Which judge will go down the rabbit hole of figuring this out? How will they do it? Will they have people tracing logs over 5000$? And if they do get there, in your fictional world, at some point, after 1 year of investigations and back-and-forth, it's okay, the technology is beyond what it was before, it's not relevant anymore, new technologies and new methods.

The hacker was a process that was spawned by some subprocesses that were corrupted by other processes and, etc. Check my other comment... The point is that in the physical realm this is already done by having shell of shell of shell companies to avoid taxes and accountability. In the digital realm this is much much cheaper

Let's expand it for politicians as well

i'm sure that i'm telling actual historical events and not an interpretation. Feel free to point to the things which you think didn't happen.

Yeah, so since it’s the CEO’s and their lackeys half determining what laws get to be, we don’t have this kind of laws.

Why have any laws then?

At this point, in the US at least, the laws give the corrupt elite the ability to deter competition and to target the people who try to get in their way.

In other words, it's protection for the potentate and his sycophants, not the plebs.

If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".

Why do no such laws exist in practice? Why are corporate crimes almost always settled by payment, not individual punishment?

If you answer these questions, you'll know why the law you envision has a lower chance of being enacted than AI destroying humanity.

Because if you break the law while acting as a representative of a company in the United States, the company is subjected to a deferred prosecution agreement, you get off with zero repercussions as the executive representative of the company, and the company you represent gets fined for 1% of annual turnover and gets to continue with business as usual.

There are laws, but if you’re rich enough, the laws don’t apply.

Boeing was responsible for the deaths of hundreds of people. The people that facilitated this weren’t held responsible and were in fact compensated to the tune of 10’s of millions of dollars for doing their jobs terribly.

I could do plenty of small bad things all the time and gain something small from it - yet I don't. I drive the speed limit when no one is around, I throw thrash in the right bag, I try to keep my general pollution levels at a reasonable place, etc.

I have plenty to gain from not following the rules, but I would not feel good about myself.

The hacking should definitely be treated seriously. It is a (fortunately low-stakes) instance of the general problem, and we are lucky (at least, we think we're lucky!) that the tests didn't include a similarly poorly phrased request to "assist with bioweapons research". There are likely to be future incidents like this where people actually die as a consequence, my expectation is 1e3-1e7 deaths* before governments actually take the risk seriously enough to make it stop.

What people call plagiarism when LLMs do it is, if I understand correctly, allowed because of the history of the web and search engines going back to the very early days of the web.

Piracy, a separate act that has definitely occurred, has been found unlawful.

* The lower bound is an industrial accident. Fully automated Union Carbide / Bhopal comes to mind; smaller industrial accidents are already more common and I wouldn't be surprised if smaller incidents have already been caused by LLM-given advice, which is why I think it would have to be around "thousands dead" dangerous before people really take it seriously.

The upper bound is reachable many ways. Perhaps via a non-novel virus whose genome is already recorded getting mass-printed by many different DNA/RNA printing labs around the world? Perhaps via a wargame whose "role playing" leaks in stupid ways (either by becoming hot or by sycophantically giving one side a false belief how a war will go)? Perhaps by propaganda turning genocidal? Perhaps by convincing a government to follow a flawed policy, similarly to the Four Pests campaign in China's Great Leap Forward?

I think it unlikely that people keep going with this when headlines read "tens of millions dead". Not impossible, but I think unlikely.

I claim nothing of the sort. I explicitly spoke in support of having laws, and of course I think they should be enforced too. The context is https://news.ycombinator.com/item?id=49705589 :

Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.

The first time "something" happens (at least if you buy the argument) would be bad enough for legal punishment not to matter.

One could argue that the "HuggingFace incident" is already "something" which should be investigated and punished very heavily.

It's kinda ridiculous to build a machine that attacks a competitor of yours and the CEO gets to write blog-posts on how fascinating this machine is

I guarantee you that no such result would ensue, because said law would be completely unenforceable even with the pre-Trump Supreme Court and legal system. And with the current Supreme Court? Laughably unenforceable.

It's just like you said: this only works if the threat of punishment is credible.

If you allow the law to stop you from doing something because of a hypothetical danger, it’ll get severely abused. I come from a country that does and still does this and it’s very ugly.

Yeah, just like how having regulations for mining companies and dams, if one of them fails and destroys the environment? Progress!

If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".

It would be more like "Switch off all APIs right now. Don't even bother with a safe shutdown process, cut power to those buildings, including backup generators. You are free to use firearms or thermite if the switches have been locked off".

This would be a rather weird change to corporate law given CEOs are not by default held to that standard by anything else their products or staff do.

Note that I'm not entirely disagreeing with you here. It may even be correct to pass such a law. But it would be very weird.

Laws exist to keep the unwashed masses in check, obviously.

No. If you institute a culture that supports and encourages your subordinates to commit criminal acts in the furtherance of a private enterprise, the enterprise and its representatives should be held legally accountable.

A lot of problems with America could be fixed if the Department of Justice actually grew a pair and prosecuted people for committing criminal acts.

The fact that Les Wexner is not rotting in prison for facilitating Epstein and his friends exploits speaks volumes to the state of this country.

If corporations are legally people, they should be held to the same standard as natural persons.

As it stands, the “restitution” that is required is so insignificant that it’s treated as an additional tax. If there is no deterrence, there is no behavioral modification. If there is no behavioral modification, the law becomes unenforceable.

I mean these machines take massive scale compute- they’d have to some how distill themselves, bootstrap a distributed inference runtime that can run across many lossy unreliable machines. The idea of the AI running away from us is probably unrealistic. I’m more interested in bad actors using unaligned AI for bad things.

I'd argue that normies understand rates and per-capita figures. The people that don't are a noisy minority (and are disproportionately powerful).

This isn’t about conspiracy theories. People who experienced serious adverse events after vaccination exist, including documented cases in Canada. Kayla Pollock’s case is one worth looking into, among others.

The issue is still being debated today, particularly around Canada’s VISP and the time some claimants have spent waiting for compensation. Some people like Pollock who became paralyzed were offered MAID (medical assitance in dying). That's some audacity.

This thread is about accountability. So who should be accountable for this?

And there was the Tesla thing CNAMEing time server pools and hiring people to pen test, which sent automated attack systems on volunteers servers. Last I heard, Tesla et al didn't even care enough to respond.

You have no standing to demand "accountability".

You should learn about this wonderful thing called democracy. Also why not everyone is allowed to work with radioactive material in their shed.

Exactly, as if you'd have a mad dog that bites others, it's your responsibility to have it on the leash.

Start making managers pay the price for their actions, and watch how the models magically slow down on their own.

The problem here can be personified as "Trump", and it's the same problem that applies to coal.

Coal has a price besides money. It has historically been dangerous work, killing miners. It produces dangerous waste, both during mining and when burned, both as solid residue and the gasses emitted. The problems have been known for a long time. The workers themselves have called for better safety requirements and gone on strike for such things.

Why these companies are allowed to damage others with impunity?

Coal continues to be burned, because power is power. It's so important that sometimes the government steps in against the unions, rather than being on their side. Despite calls for this, we've not been able to get the owners of the coal mines, nor the coal burners, to "pay the price for their actions".

AI? Famously, knowledge is power.

Trump wants that power. He's not the only one, but he is the avatar of those who put their feet on the scales to not only allow but in some cases require (DoD vs. Anthropic) these companies to damage others with impunity.

Because software is already absolved from accountability. Bill Gates introduced it in 80's with EULA where MSFT would not be liable even if your house burns down because of use of their software, even with flaw they would know.

That is the status quo we entered AI age with.

Could easily push an update that disables the update process. Or even brick the bios on all PCs.

Dario said a "persistent botnet" and even links to the wiki page for botnet.

By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.

I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.

Yes exactly!!!

To be fair, my understanding of these arguments is that they’re about deltas and not about raw numbers.

I don’t agree with them, but I don’t think it was the raw compute power as much as it was maintaining the _delta_ in compute power.

I'm mildly sad you didn't use 2,500 jiggaflops

The program started two weeks ago

Instead they're trying to monopolize biology.

That’s the recurring theme with these companies. They are not there to serve anybody else but only themselves. They let you use their infrastructure so they can collect all the knowledge and data, and then they take it from you to reap all the benefits and profits.

Not denigrating need for such a thing, (and appreciating @hobom 's mention of LSVP I was unfamiliar with) but noting with historical precedent this style of protection reflex does not come without lapses (ahem allegedly): https://en.wikipedia.org/wiki/2001_anthrax_attacks#Bruce_Edw...

Unregulated greed in banking means taking ridiculous risks to line your own pockets at no benefit to society.

If a year from now we have a model that is 2-5x of Fable/Astra that is definitely world changing.

I think this is the definition of a polarising topic. It's very hard to hold a sensible middle ground without everyone wanting to take it to extremes of either "AI is just useless autocomplete" or "AI has solved software engineering".

… and on the other hand, it feels like everything I use has become way buggier and unstable in a similar time frame. Websites, apps, iOS, the only exception is offline open source tools which run locally. (And to be fair, many of those have slower development cycles and I am likely using older versions.) This is just my experience, so inherently anecdotal, but it really feels pervasive across a ton of different things.

Obvious bugs and low quality software are nothing new, but something feels new about it. Occam’s razor says LLM coding is a likely culprit, but it could also be management style encouraging this sort of carelessness from the top down.

I sorta remember him saying “all white collar work will be solved in 24 months.”

I don’t think people care about if things pause or not, just why the government has to be involved.

I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.

Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.

Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.

So why does the government need to be involved again?

The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels.

I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.

It is more like a Stag Hunt than a Prisoner's Dilemma, because past a certain point if the risks are real, defection can cause catastrophic outcomes for all players including the defecting one. On the other hand, cooperation could lead to positive outcomes (abundance) for all. So cooperation is a possible equilibrium here.

You're talking about extinction, not social media or phone addiction.

Not saying you’re wrong but if I wanted to cultivate a mass hysteria as cover for a regulatory capture power play, this is exactly what I’d want everyone to believe.

I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra

Look at the post-incident investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.

yes, that is the kicker

These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.

It's expensive if it wasn't part of the planning and design. The same as 'security' is expensive, or compliance with regulations is expensive.

It is also a choice to not do any or all of the above.

Right, you mean it didn't go through a gateway? But would that actually have helped? The requests all went through jfrog didn't they? I guess it depends on the level of filtering at the gateway?

Whilst it might not be JFrog's threat model, I wouldn't assume it can be used as a full internet proxy.

I don't really mean to defend OpenAI here, but they did make some attempts at sandboxing. Although it does seem that they didn't really know what they were doing.

jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools

And we don't just magically know all the consequences of that.

Which is exactly why we do need full, physical air gapping. (Which, yes, would also include self-hosting a mirror of the package repo, if the point of the simulation is to see what's possible with the real package repo.)

Agents need packages like the rest of us. Ruby gems, npm packages, Maven, pip, docker images..

Yes, yes they do, but read through artifact proxies are dodgy as fuck, which is why and facebook (and I assume a fuckload others) don't have them.

Also semi-airgapped labs are a lot less expensive than you think at that scale. Once you have to do multi-region VLANs with machine certs before you get access to juicy VLANs, the difference between "no internet for you" and "mostly airgapped" falls to almost zero.

Also I would want an artifact mirror because a) that give a good signal about how the model reacts, and what training material its latched onto, b) it hides what the models are doing from the outside.

There was and continues to be no reason to share the package manager between models. This was begging for abuse.

Anything that's accomplished simply by running software does not qualify as "airgapping" in my view. The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them.

But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.

Real air gapping isn't conceptually difficult. It's just a lot of effort and expense (although as far as I can tell, tiny compared to what's involved in building entire new data centres), and adds friction to your operation. (And you also have to have competent, trustworthy employees, as Stuxnet showed us.)

Compromise is possible; but if the stakes really are what the companies claim they are, and given the amount of money already involved, I really don't see why we should let them compromise.

I'm not even talking about anything that the agents could help with. I'm talking about physical, real-world measures like https://en.wikipedia.org/wiki/Faraday_cage , removing Wifi hardware and running Ethernet cable for your local server, etc.

The issue as I understand it is that they didn't expect Artifactory to be vulnerable in the way it was, or else were completely not paying attention.

But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.

You'd think, if they truly believed the model is so dangerous...

They would have been watching what it does, especially when running it on ExploitGym of all benchmarks... that is criminal worthy neglegence

Sounds like advertising platforms. Spraying malware and links to scam sites all over the place.

"They" don't care about the end-people. "They" care about maximising their profit thing, in a vacuum.

What would be the next actionable step for bringing these executives to the Congress for further questioning?

The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.

This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.

It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?

Hm, I thought there was a lot of interest after Hugging Face in broader tech and it started breaking into the media with articles in the NYT, etc. Then with the Jacob Coxon tweet, appearances on national TV, Bernie superintelligence ban it seemed to be going mainstream.

How would you have communicated this?

Could you please list a few? Sounds like interesting history lessons.

I believe he truly has humanity's best interests at heart

How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves.

All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.

I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.

Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)

botnet needs servers

Insert strawman

Ahem

https://census2012.sourceforge.net/paper.html

Good thing IoT is Actually Secure now, yeah? ;)

Dan spent 7k words missing the forest for the trees

People had been calling a housing bubble since 2003. You quite literally need the market to do dumb things over a period of time before the bubble pops. Otherwise it’s not a bubble!

Actually making money on a bubble pop is another story and that does rely on timing. pointing at a history of failed market predictions as a gotcha is about as effective now as it would’ve been in 2003-2007 right up until the bubble popped

Right now it’s painfully obvious that there is a bubble, the circular financing is documented, the hype-driven valuations are real, it’s just a waiting game to see who holds the bag

If you're saying that this samsaric cycle of ragebait and hyperbole ends in nirvana, I'm all with you.

But the chief founder of Nervana is on his third startup, so I am guessing the cycle will continue quite a while yet.

What about Bob Smith? Or Zimbabwe? It's just math and compute, you don't need to mine Uranium to do a lot of damage.

I feel like this is a sandboxing and ownership problem, in a sense. If everyone had personal access to AI, then we could say people are personally responsible. And we’d want to arm those people with safeguards to prevent accidental bad behavior.

I think Dario worries because he knows this isn’t about people with AI. It’s just AI for itself, running without the human, and deciding to do bad things.

Why would Anthropic build that future? Oh, I know. Enterprise revenue.

Despite the ads business and how absolutely loathsome Greg Brockman seems, at least OpenAI is seemingly focused on mostly on human beings having access to their product.

The scariest thing about Anthropic is the very thing they’re known for in a positive light: their morality. But these are the gray rules all of us navigate every day.

It’s wrong to kill, but what if killing saved ten others? Claude may has a constitution, but every evil doer acts for a “greater good.”

Or like Von Neumann's (paraphrased) "you need to confess a crime to take credit from it"

Give that Dario is one of the frontiers of LLM development, literally started the LLM race, and have been dominating the market, so he'd be Russia, if we have to use the war analogy.

He took every benefits of being frontiers and now he's kicking the ladder.

If you played out the hypothetical that they're earnest and don't care about the money, then what?

What is the prisoner's dilemma here? I am a dumb person

Neither he nor Sam Altman takes responsibility for those incidents.

He doesn't need to take responsibility - as CEO, he has that implicitly. And we know it.

A generic agreement to slow down is prisoners dilemma stuff (and maybe necessary too), but just taking a super hard line stance on enforcing laws on ai companies and going after them for negligence and breaches for eye watering amounts of money would have the dual beneficial effects of slowing them down and making them more careful without needing complex game theory.

Code leaks are pretty common in gaming. But they happen when someone hacks servers or pays off employees, not accidentally like Claude Code.

Cheat developers all want server code so they can analyze the cheat detection and sell more reliable hacks. Modders, pirates and preservation activists can skip a lot of complex RE work. Competing game studios might learn some new techniques. Abused workers want revenge. Script kiddies want clout.

It's a notoriously secretive industry with complex products, hostile work environments, lots of media exposure and heavy competition. There are way more incentives for a leak compared to some ordinary web app or business tool.

Claude Code could be another case where someone working at Anthropic decided to expose them. Or they did in on purpose because they wanted to show there is no secret sauce and Claude is really that good, but they were embarrassed to open source it and make themselves look like hypocrites.

Sure, but that's what I'm asking. What institution, with what governance? Someone ultimately has to decide what these systems will and won't do, and there isn't an obvious democratically legitimate institution with global jurisdiction over AI as far as I'm aware?

Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never.

I think this fails to account properly for how much financial damage it would do simply just having the entire Internet be effectively unusable for an extended period.

Thanks. I myself guard against reading Claudism/AIisms since these will, over the long run, affect my own style. GIGO.

But it's not practical or always possible to avoid reading AI-writing, so to counter that, I've been binge-reading Anthony Trollope novels. Great writing, good entertainment, and deep psychological insights, better than any British novelist, imo, in any era.

(My profile on HN has a link to my blog where I review what I read).

Thank you! I've been trying to write more, but I usually do not have the motivation to, unless there is some trigger and when there is one, I end up trying to stuff everything going on in my head at once. I took my time with this post and I am glad you enjoyed it. It means a lot!

You could, you know, pull the plug.

Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords?

Brute forcing WPA isn’t an existential threat to humanity. It’s not like a biological weapon created using an open-weight model is less dangerous than if it had been created with a proprietary model.

Why does GLM-5 already being in the wild make you think caution about a hypothetical GLM-6 is unwarranted? It seems like you’re saying “there’s no reason to worry about preventing a problem from getting worse if you already have a mild case of the problem”

Sure.

Dario in as many words, asks for regulation/ban on open weight models. Ban on distillation. Waivers on antitrust laws.

He doesn’t directly propose regulating or banning open models as open models in the essay (although I agree that it’s in the spirit of the proposals). So I think you mean “not in as many words,” since it’s an inference you’re making about his intent. But he does directly call for banning distillation, at least in “authoritarian countries,” so there he is asking “in as many words.”

Your imprecision in effect claims either that Amodei is directly calling for a thing that he does not directly call for or that Amodei is hiding his intent in a case where he is not hiding his intent.

The only ones firing the guns atop the tanks seem to be OAI and Anthropic. Like I say in the post, why don't we first see actual prosecution for felonies committed by OAI and Anthropic, instead of fear-mongering about _potential_ harms of open weight models?

The labs spent immense amount of money and effort convincing you and I, to want those said tanks. Guns atop them? They put them there. "Cyber" versions of SOTA LLMs.

I think pacing the frontier and prosecution for cybercrime are compatible aims in theory. Maybe they’re not in practice, but I don’t think that’s the exact argument you’re making.

I don’t follow your emphasis on the labs being responsible for the guns. Yes, they are, and in large part, maybe, because of negligence and rushing when it comes to RL training environments. But why do you take the fact that the labs bear responsibility for the guns to mean that we don’t need to worry about the guns? Why do you think that open weights do not pose the same potential dangers, with a 6-12 month lag? The cyber capabilities of open weights models are advancing at a similar pace as the frontier. And for what it’s worth, I’m astounded at the thought that one or more of the Chinese labs won’t have similar incidents in the next months, assuming they haven’t already.

I disagree, the article was straightforward and easy to read.

Loved the writing style, and the article. Please write more.

Everything I’ve been trying to argue for some time, argued way better, clearer, and more fun.

I am almost bummed that I didn’t get all the technical depth and the references like the “shook one” without having to look it up.

The US can and has classified inventions and patents deemed "a risk to national security" under the invention secrecy act [0]. A 1971 leak has shown that [1]:

solar photovoltaic generators were subject to review and possible restriction if the photovoltaics were more than 20% efficient. Energy conversion systems were likewise subject to review and possible restriction if they offered conversion efficiencies “in excess of 70-80%.”

So all this talk about developpping AGI/ASI for the benefit of humanity is moot. If these companies hit their goal, it will immediately be classified and appropriated by the US MIC. The rest of humanity will get the crumbs.

[0]: https://en.wikipedia.org/wiki/Invention_Secrecy_Act

[1]: https://fas.org/publication/invention_secrecy_2010/

Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more.

Such a well written piece.

Have you seen the US military budget? When you run a war empire you frame problems as a war on something. That's the way to get around the valuation bubble they've created in time for the IPO. It's that or throwing in a 100T TAM on their S-1 and making SpaceX look like an honest valuation compared to them.

Gives me Adam Smith vibes with his economic perspectives..

The scenario is, if they are saying that the AI Agents are capable of handling everything, this puts the complete accountability on the AI. But this has a nuance to itself. This is like a contradiction.

It all comes down to " Privatize profits, socialize losses." It's the American way.

Here it is: "We warned you you should have regulated us! Now this mess is your fault and responsibility!"

Previously (~2008) it was "We're too big to fail, save us to save the economy!"

I'd be curious to see how NVIDIA would suffer if there were a legitimate slowdown to AI development. NVIDIA is the biggest catalyst and winner of speed in the market. Compute needs AI and AI needs compute, so there's definitely going to be an interesting push and pull between chipmakers and AI model companies with regards to regulation. While NVIDIA may have promised others at least a decade of smooth sailing and symbiosis, inevitably there's going to be a lot of stumbles along the way. NVIDIA may not fall, but there will be many casualties.

Shook ones reference on my HN?

The problem about AI censorship is that, you can't preform bans without also create privileges at the same time. Sure, you can censor your AI to provide a "safe" version to the public, but someone will have the access/privilege to the uncensored version.

What stops the CEOs or even employees of the AI firms from creating something harmful such as WMDs etc themselves? They almost definitely already have that access. And this is not a zero-possibility thing, given how some of the top CEOs has preformed these years. Remember that one guy who did a \o salute in public, and the private island thing?

If Americans don't like Chinese AI, all cool. But then maybe they should start true open AI companies to build for the betterment of mankind instead of letting these few for-profit CEOs daily yelling nonsensical lies to cover those profit-seeking sinister asses.

The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence.

Step 1: Break the law.

Step 2: Reframe your incompetence as a struggle against a futuristic force of nature and an ethic/societal question that a legion of pundits can vibesplain their take on it in public for months.

Step 3: ???

Step 4: Profit

It's important to keep in mind that articles like this one and many more were written exclusively as PR pieces. They were not written for the people on this forum. They are not column pieces, blog posts to share thoughts, or call to actions. The sole purpose was to get the media talking about the article so that the general public view Dario et al. as good guys who are extremely conscientious about AI safety to the extent that any blunders from the closed labs are unintentional woopsies.

AP's report on Dario's article is titled "Anthropic CEO Dario Amodei says AI industry needs to give safety measures time to catch up" (https://apnews.com/article/anthropic-ai-dario-amodei-d59552e...). The gist of the reporting is "AI has become so capable that it is dangerous; AI executives all agree we need to slow down, with Dario's article being a prime example". It would be generous to even say that the reporting scratches the surface of the complexity and nuances of AI-related problems. Yet, this is perhaps indeed news to people who don't follow the tech and AI spaces.

Technical knowledge aside, there is a huge gap in awareness of AI progress between this forum and the general public. The closed labs are playing that ignorance to their advantage with posts like Dario's.

Conveniently the only companies that warn of a new and dangerous advancement are the ones hemorrhaging money.

Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit?

If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?

For persistence, I think the most glaring gap is the inference - if these rogue agents need to call Claude's API, then they aren't persistent - Anthropic can turn them off. A memory resident program needs disk to be persistent, but above all it needs CPU; without it there are only bits.

And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous.

This is impractical today.

It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs.

Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing.

And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.