Link for those who didn't get the reference:

https://x.com/AlexBlechman/status/1457842724128833538

Chinese companies just use massive government subsidy

[citation needed]

Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.

Yeah totally unlike the US where tech companies are awarded multi-billion dollar contracts of taxpayer money in no-bid contracts without oversight

And where the Vice President has almost exclusively only been employed for the primary investor in several such companies

Meanwhile US models just use massive investor subsidy to distill American copyrighted work and rent the models to everyone for money. That's true capitalism. /!s

I completely agree, but:

List of fraudsters pardoned by Trump

https://www.businessinsider.com/list-billionaires-businesspe...

The current attorney general and former Trump lawyer on Fox News yesterday on AI (sorry, "super intelligence"):

https://bsky.app/profile/atrupar.com/post/3mwissj34rt22

Anthropic's Dario Amodei to have White House dinner with Trump on Sunday:

https://www.axios.com/2026/09/27/anthropic-trump-dario-amode...

77 million Americans brought this onto themselves, and all of us.

Can't wait. They've been criminally negligent in most jurisdictions. Trump can help you outside the US.

You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?

If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.

So powerful they can't even do math properly without a staff worth millions writing all the clutches so that they can, wooooowooooooo

And they are not calling it a day and they have done most things possible to communicate the fact that they are building something dangerous.

It may have been novel for OpenAI but we already had Mythos at this point and this talk by Nicholas Carlini.

https://www.youtube.com/watch?v=1sd26pWhfmg

We already knew LLMs were capable of finding exploits like this.

Unfortunately, they're controlling the narrative. It's unlikely that their negligence will come to light, given how many powerful entities are invested in their financial success.

Artifactory is not a hardened piece of software meant for adversarial workloads like the one OpenAI was using it for. This is just common sense. The sandbox they setup was like putting their AI in a jail but allowing it to leave the jail on its own to go to the convenience store.

The most recent DNS sandbox escape is similarly ridiculous.

Lot's of hints in the Wikipedia page: https://en.wikipedia.org/wiki/OpenAI%E2%80%93HuggingFace_inc...

E.g. even the basic thing (block the internet), was not actually properly blocked.

Some comments: https://techcrunch.com/2026/07/22/how-an-openais-human-mista...

This. If an AI can't be deployed sandbox-free, with little to no supervision, without risking an oopsie? Then an AI oopsie is inevitable.

Practical AI deployments aren't going to do ridiculous bullshit like "airgap the server farm" or "route all inputs through a data diode". They'll give an AI root access on production servers so that it can run diagnostics live during an incident. Then they'll forget to revoke that access.

If an AI can't be trusted not to take malicious actions in pursuit of its given goals even if deployed in the most half-assed manner and given more than enough access to take those malicious actions, we have a problem. Evidently, we have a problem.

Even if what you're saying is true, did they monitor the outbound connections from the training network? Was that a coverup by the bots too ?

Seems pretty wild these things were hacking government websites etc but yeah no one picked that up until the victims reported it?

You read right. For example, using Artifactory the way they did (unmonitored live proxy mode) was pure negligence + laziness/incompetence. Especially if they believed even 10% of the "imminent runaway risks" they had already been harping on for months. On top of that, no (or at least entirely insufficient) monitoring and human oversight. Even after they had previously been hit by the same class of "sandbox breach" multiple times, as GP alludes to.

this happens all the time. have you heard of the Sackler family and the opioid crisis?

the difference here is that they’re calling for someone to stop them, which is both weird and unconvincing because these immensely powerful billionaires can in fact make their own decisions

If I have a random number generator and I also have the burning desire to connect the random number generator to the nuclear launch system it is both just a random number generator and also I should be tackled to the ground by serious men with sunglasses in black suits and sent to prison.

OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.

Were their sandboxes in-process with the harness? Were they actually better than something like bubblewrap or even docker?

When the Onion is not the only website that publishes satire, what’s that a sign of?

This is a satire website too.

"said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff"

"Anthropic shares skyrocketed upon the revelations; a remarkable development, particularly given the company is privately owned."

Perhaps it's mocking the misleading press coverage of AI companies?

It is a parody site.

OpenAI CEO Sam Altman celebrated the breach as an “alarming threat to cybersecurity.”

And even more dramatically, when asked about rumours that its model, Claude, had killed his wife by hacking into the family’s WiFi-connected microwave, Anthropic CEO Dario Amodei replied “Well, yeah, sometimes.”

I had to read that twice. "Oh okay. The whole thing was satire."

Musk was told by Demis Hassabis in 2012, see https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...

Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1

I also read that book in 2014, and it is how I came to understand this problem early.

You communist, on the other hand, would prefer it if a Maoist AI once again exterminated 60 million people.

BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.

Quick search does suggest to me that a number of - even prominent - abolitionists were slave owners, but I'll be honest, the topic is hardly my specialty.

As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.

Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.

Falling behind on useful AI capability is obviously not in the interest of anyone

Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.

This was fun games until their fictional machine god started actually getting very intelligent.

If you all believe in the same prophecy and hang out at the same orgies together, I'm willing to call that a "cult", at least in the colloquial sense.

Would you at least concede that the Zizian offshoot was a cult? They murdered multiple people!

The only thing that can stop a bad guy with a god is a good guy with a god.

OpenAI claims to have paused the training of their ”most capable” models: https://apnews.com/article/ai-openai-anthropic-agents-rogue-...

Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true

The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.

Project 2025 is a great analogy, because of course, they did exactly what was in the plan despite distancing themselves from it during the election.

When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.

Wow, you're out of touch with how LLMs, harnesses and agents work.

This is the AI CEO equivalent of saying your biggest weakness is "working to hard".

they need to buy the AI-stock for their safety.

But this is wierd: How shall I convince one to buy a product if I say "it-is-dangerous"? :-D

Like how Tony Soprano sold sandwiches and ‘protection’ at the same time. Two income streams are better than one.

Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm

You can download some weights and run your own model on your own device, and apparently only be a few months behind SOTA. While things are cheap, people will pay for the premium product, but if they ever try to crank up the prices, there's a fallback that is quite competitive.

As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.

I think the current state isn't what the frontier labs are worried about. The real issue is the dynamic that open weight models create.

The frontier labs are investing TONS of capital into training the new 'best model ever'. Today, that doesn't make economic sense, it loses money. So where's the business? The future, where token costs reflect the costs involved.

For that to be corrected, the labs need to get people using the models for things they depend on, then they turn dial up on cost. The ordering there is critical. It has to become indispensable first, then costs go up.

The trouble is, open weight models are rapidly becoming usable in what is looking like the big market: coding. Step 2 (the costs go up) doesn't work if there is a ready off-ramp that doesn't require frontier models.

However, if they can effectively make open weight model development illegal (its dangerous!), their business strategy can remain intact. A nice side effect is that they will be effectively be granted an oligopoly "for humanity's safety".

If they can convince the populace that a text generator is an existential threat to humanity, that's a hard argument to combat.

Edit: I think it is ridiculous that autocomplete on steroids is an existential threat. There's no way anything harmful happens until you actually use it as part of important systems. By that reasoning, you could make it illegal to use AI in important systems, but do you think that's the argument the frontier labs are making to governments?

That's an accusation others are making:

https://news.ycombinator.com/item?id=49875725

But it doesn't live up to the promises of monetizing it.

Realistically, each new, better model will only capture a smaller and smaller part of use cases for which existing competing models were not enough.

So in few years (arguably, already started) they are in race to the bottom for service to majority of their customers, and with companies that have far smaller R&D bill.

at the same time, more and more models will be ran locally, and that eats into potential profits of them too

On side note, I wouldn't be surprised if GPU vendors started making DRM for AI models so you could "license" someone's model to run locally for x mil tokens per license...

But issues is that for the valuations you need models that work without human expert and Opus 5.5 isn’t there even easy stuff like programming.

Let’s all pause for a moment and really take that in: these major enterprises are choosing to go to the press/public to acknowledge their felonious conduct because they need to in order to get more capital. POSIWID and all that.

The Dario bit on SNL’s Weekend Update this past weekend was on the nose.

What is even more funny ... AGI actually makes everything worthless. (I don't believe we can achieve AGI with current means to be clear)

Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".

But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?

they’re confident that the current administration won’t act between the direct payments and how much money they have riding on American AI supremacy.

I just don't get this though. It's clear there's no major difference between foreign models and US models, they're not actually in a different class, the US ones just have more resources at the present time and a bit of a head start.

If the current pathway is a pathway to AGI, everyone else is on schedule to also realize it, maybe just 6 months after the US gets it.

So... what's the plan then?

either liars

They're CEOs of tech companies with insane valuations

or homicidally reckless

They're CEOs of bleeding edge tech companies with huge capital and military applications

So, they're either liars or homicidally reckless.

Yes!

Why not both?

Unfortunately if the frontier labs that care about safety stop or slow down unilaterally, that doesn’t make the problem go away. It makes it worse when labs that do not care at all about safety, or deny that it is even a problem, are leading the way. That’s why there’s needs to be some form of international regulation.

My guess is its a combination of both. Either way I'm not a fan.

Hallucinations come up when asking knowledge bases questions, such as "In React’s Canary Fragment refs API, which FragmentInstance method returns a flat array of DOMRect objects for all children?" Or "Over what years did Roubini and Sachs examine 15 OECD countries when assessing trends in tax-to-GDP ratios?" While to me those may seem hyper-specific and unlikely to come up in a real context, students will absolutely ask questions similar to these.

Are you suggesting that it's higher or lower?

IME on real projects you do need to be very careful with prompts about topics that are less likely be common in the training set.

I don’t believe that they are truly concerned about much beyond their own self interests.

They believe it's real and they MUST be the ones to control it. That's how they keep their self-interests aligned.

Fun fact: 'they' are human beings too and some of their interests do allign with yours. They have kids too and might want them to have a proper future, they don't want to fall I'll to cancer either etc

the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks

You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.

Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.

The call to slow down is because politicians are now receiving less money from these large corporations because it's being burned om AI and data centers.

Thats my theory Preventing competition to stopp the race to near zero pricing

There isn't enough compute for cheap models to destabilize the giants.

Back in May, Google was serving daily what openrouter served in a month, all models combined.

Compute is the moat.

It’s more to get you thinking about them, talking about them, writing on the internet about them…

the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain

I think this actually makes it far more likely something unintended does happen

I agree with a lot of this statement, but I would argue that this has been the case for a long time. See the giant list of social startups

Rich people are mostly retards who had rich parents. What now?

they have completely failed to deliver all the promised impacts of AI

Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?

"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.

I think this speaks more of the average investor than the average CEO.

It sums up how Corporate interests are basically on the same level of US police forces "No requirement to protect the public" and the same "qualified immunity".

If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.

scheming about something else like regulatory capture to protect their market.

This but they also equate dangerous with powerful, which is just marketing again.

I agree. AI lowers the bar. Anyone can be a cyber criminal now whether they understand what they are doing or not. I think that's the danger, and that's not just limited to cyber security. People can now get help to do things (good or bad) that they could not just a few years ago.

Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.

Exactly! Always for the "greater good".

A caveat is that in this case third parties have plenty logs of the bad behavior.

If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.

There are definitely economic incentives behind this. I’ve blogged about this recently [0]. My guess is the main reason behind this is that current SOTA models are already too expensive so it doesn’t make economic sense to train even larger models. Only exception is government paying for exclusive access to those.

[0] https://www.vincentschmalbach.com/economic-incentives-behind...

Random thought… I wonder if the goal is to get it classified as a weapon of sorts, then it gets export controls put on it, perhaps forcing government to take on all that debt.

They would recover the training cost eventually if they stop training

"if they stop training" is exactly the point I'm making. They can't stop training if the competition is still training.

It makes me think we havent heart of the worse types of hacks the models are pulling off.

Well, it is obvious;

They know they will be favorite govt boy once any regulations hit and they will hamper competition far more thanthem

And "it's so dangerous govt had to intervene" is just advertising for IPO

It is weird but I try to resist the conspiratorial take.

It may be as simple as:

* they believe they’re going to make trillions no matter what

* they’ve seen tobacco and what’s coming for oil companies and figure nobody will be able to say they hid the dangers

* they see a 100+ party prisoner’s dilemma and know that someone will defect so the optimal strategy is to defect

To me that seems a lot simpler.

If you have the RAM and you use models like Qwen4 (3.8 flash next) that can fit 6 to 14 262k contexts in 48GB of system ram as cache.

Yes, exactly. To spell this out in no uncertain terms: for some local setups (sufficiently large VRAM, and sufficiently low concurrency) cost of "cached input" is as close to zero, it might as well be literally 0.

It's got two purposes I think. One is the obvious regulatory capture. The other is edgelording, which gets a lot of attention and makes the product seem like it's more powerful than anything else.

Meanwhile, users don't seem to care one iota.

Of course it must be dangerous - how else would you pitch it to the military?

Well, or at least by The Civilian, a New Zealand version of The Onion.

Here's a relevant article by the Onion from the summer of 2022, before ChatGPT. https://theonion.com/ceos-warn-against-the-dangers-of-artifi...

pretend, not demonstrate

This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.

this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones

It's especially funny because they are the people that the article is making fun of.

The Civilian is the NZ version of The Onion.

It's public knowledge that the code is 000000 because the risk of the president being incapacitated making it impossible to set the bombs off was judged worse than the risk of accidentally setting them off.

You're forgetting that those fences were made of paper.

The truthers will be the first one to say I told you so when agents do another dangerous thing autonomously.

For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.

If James Watt had gone to the government and public saying how dangerous the steam engine was, and how many people would perish in boiler explosions if he wasn't given exclusive control over all steam engines henceforth, he would have been rightly called a swindler and laughed out of the room. He would have been correct that they can be dangerous, but utterly wrong to suggest that warranted him having a personal monopoly or oligopoly.

And lo and behold, nobody even remembers their promises to IPO this year. Can't now, it's too dangerous!

A recent Saturday Night Live clip featuring an impression of Amodei that hits on this theme,

https://www.youtube.com/watch?v=-Nvne3LzBls

Just read the tone of this article:

said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff at Victoria University.

Don't we need more humorous yet serious writing like this in the world.

Honestly, I'm running out of ways to express what Sam and Dario are. I'm now down to 1 word: Wankers.

Why aren't the AI companies hacking each other?

Then we can truly see who is more dangerous?

:)

And I'm waiting for the story of Amodei sending a 6ft humanoid robot to OpenAI headquarters asking for Sam Altman.

I wish they would call their bluff. Just declare them a national security concern and send in the feds. Lets see how fast they back peddle on this. Any if their models are hacking everything like they claim they should be charged for it like any normie would be if they did the same.

I had to double check if this was an Onion article or not.

Oh dear. I appear to still be stuck in the onionverse.

Consider this "strong" premise:

    So-branded "AI" models (that is, big bags of numbers and the linear algebra needed to exercise them) can be inherently or autonomously dangerous at a scale that threatens the long-term survival of the human race.

Consider also this "weaker" form of the same argument:

    So-branded "AI" models can be inherently or autonomously dangerous, if not at a scale that threatens the long-term survival of the human race, then at least at a level that significantly threatens, inconveniences, or harms humankind to a noteworthy extent.

Here's a few hard truths that no one will like to hear:

1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.

This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.

2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.

3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.

Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).

4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.

The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:

- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?

- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?

- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?

- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?

Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).

So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.

At this point, AI companies are basically competing over who can destroy humanity first. What a time to be alive.

There are modern legal frameworks around pet liability (dog bites, "vicious" breeds); premises liability (e.g., swimming pools) and known hazards that stem from ancient legal concepts of whether an ox was a "known gorer" or merely involved in its first goring incident.

Some of these LLMs are known gorers.

Even the models we have access to can do basic coding tasks really well, in a way they couldn't a year ago.

I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.

They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.

Self regulation won't help

You got it

Kind of makes you wonder why they built the tech in the first place, I don't know if we're on the eve of the apocalypse or not, but why the fuck build it all and then run around telling us how dangerous it is when...they were already warned?

Bit strange no?

I'm a bit fatalistic about it all now. The cat is out of the bag, unless we see some never seen before leadership from the current US admin there is almost zero point in worrying about it.

holy heck, thank you for the moderating dose of sanity. HN comments feel like a death cult lately. All I can think is that this is intelligence addiction speaking.

I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.

But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?

Well are they going to stop developing the tech or not? If not, then you can see why people don't trust what they're saying. I mean if it's that dangerous, why not step down from your role as CEO to demonstrate how seriously you take the threat? Why not immediately call a pause on all training, today ?

it’s satire…

Its like the onion.

FWIW, the site's "quote of the week" looks entirely nonsensical and made up.

If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.

Aha, so maybe the AI labs have superintelligence and just don't want China to know? And that's why they are playing these word games now?

Super intelligence won't accelerate time.

I do think something does happen/change when we reach superintelligence.

Super Intelligence will do what, fire nukes? North Korea, for example, will continue to exist very close to how they are currently after some country develop superintelligence.

Imagine if the atom bomb was created by private companies about to make their IPO.

lol

That's why they have their LLMs attack everybody. They will be way too busy repairing their hacked web sites to ask that question. Also, once humanity is successfully destroyed, money will be obsolete anyway.

I remember when HN scoffed at the idea of Mythos hacking anything.

I swear so many people on HN are developing psychosis.

The much simpler solution is that they are building superintelligence, which they think can be used for great good (like curing most death and disease), but it is also a powerful capability that obviously needs to be regulated and is at least somewhat a matter of national security.

But nooo! Surely they are building.... terminator bots!? JFC.

Data entry and administrative support, tier-1 customer service and telemarketing, basic translation and localization, content and SEO writing, junior financial and legal support, retail and routine logistics, ...

If this approach works with AI companies, why the same doesn't work for nuclear power plants? :)

It amuses me to think that there could a noble aim: they're overdoing it partly to raise awareness of the real dangers. It's something we get as a side effect of their desire to achieve regulatory capture.

Somehow, not the Onion.

My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.

It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.

They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.

The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.