Also, "rouge" is one of the oldest spelling mistakes on the internet. ALL BLUES ROUGE LFG WC/RFD

A cabaret AI would certainly be better than one trained on the Khmer Rouge.

It's carmine all the way down.

Kimi / open models or jailbreaking frontier models. Your recollection of the Anthropic refusal isn't quite accurate, cyber hacking wasn't a sticking point, just domestic drag net surveillance and fully automated weaponry.

Claude said they don’t want their models used for mass surveillance or autonomous killing (?). USA frontier AI models have been used extensively in the Iran conflict and beyond

I think it was mostly a matter of "how much" and marketing rather than "my principles won't allow for this!"

what do you mean? they're using AI to kill people directly in Ukraine

https://www.nytimes.com/2026/08/24/world/europe/russia-drone...

They don’t need to run their own DCs, just pay for a proxy somewhere in the world that has better access to the infrastructure. We know North Korea has been doing that in the US since years now

It’s an example of potential use by bad actors.

Um... China?

No one is attacking China

I appreciate this line of questioning. It's really interesting to see how many excuses people need to reach for to avoid the conclusion that the "secret unheard of power" is B.S...

The Chinese models are benchmaxxed to hell. They aren't as good in practice as all the hype suggests.

There are several ways to get hold of tech and of any other thing. Russia did not have access to the H-bomb too, until it was handed to them. Ideals or money are very effective.

Yes.

I again am just shocked the sky is not falling, when thats the sales pitch.

Day to day life in what country? Do you live in Ukraine? Maybe things are wild there and we just don't know about it.

Only difference is the height he fell from.

So what I'm saying by saying that he fell out a window is that Putin and/or the FSB arranged his death. It's not a statement of the means of his death, but who arranged it.

Because they dont have the money for hardware or compute obviously.

Please see other responses, I would expect to feel the effects not just read about.

Here's a lower bound on what misuse is happening:

https://www.anthropic.com/threat-intelligence-report-septemb...

Is the Kremlin technologically useless?

Clearly, look at what is going on with Ukraine.

They are great at propaganda, so is China, Iran and North Korea, it's why everyone runs around spouting such stupid nonsense...

Cyber attacks between these countries were happening on large scale since the beginning of the war. There were several huge breaches, but otherwise most high-profile companies adapted and significantly strengthened their protections. Rest assured, you can be sure that both sides right now utilize available AI both for attack and defense.

Ah damnit, you beat me to it. Excellent sense of humor, friend :D

If anyone is confused about this comment and similar others, the original title had "rouge Agent" instead of "rogue Agent."

Also whoever monitoring these agents, in this case not monitoring. This "Who is responsible" dilemma is so stupid. If I gave the AI tool means to kill a person but I did not tell it directly to use it and it uses it anyway then I am responsible for it.

Reckless conduct as applied to slow moving corporate planning is an interesting idea but do you really want to go down that path?

It basically means anything bad that happens is a criminal indictment against everyone who created the conditions. Have you ever released any software that anyone could misuse? Left a car unlocked that someone could have stolen and killed someone with?

To me this sounds like a recipe for selective enforcement using bad outcomes as leverage. Sure, get the AI CEOs now, we all hate them and they’re jerks. But the tool would be so much more powerful than that.

The CEOs. They have full control

They lost control long ago.

Why not both? (all people involved)

Cool. If they’re taking literally all of the risk, should they get literally all of the rewards? Nothing for shareholders, nothing for employees? Or do you want capital liability for less than complete benefit?

‘It wasn’t us, it was a bug in the software’ used to be the defense for bad code. Then it became the defense for self-driving cars. Now it’s being used for AI cyber attacks.

No, I remember being threatened by OpenAI and then Anthropic (and now both) since back when ChatGPT was seriously useless.

> AI is dropping out of the spotlight

spit take

Don’t forget outright intentional.

Both? I’m not sure what distinction you’re trying to make. It was completely irresponsible and likely a felony

nah, CEO-s have been getting a legal pass since the invention of corporation exactly for the purpose of being "unaccountable"

check boeing for example, a rather blatant example

AI is literally state sponsored so I don't see that happening unless the AI turns against the sponsor.

Wait until OpenAI or Anthropic exploit FAANG.

Proof that the AI alignment problem is hard (perhaps even unsolvable). These labs clearly did not mean to send their agents to hack RubyGems as a side-effect of testing a web scraping agent under restrictive conditions. How can we hope to build aligned AI if they consider solving their trivial evaluation task important enough to hack external systems?

Sounds more or less like the last breach then.

Doesn't rogue in this context imply "outside of set limitations"? And then not "failed to properly instruct"? The same applies to humans when given bad instructions.

which is misaligned with OpenAI and humanity

OpenAI is itself misaligned with humanity, as their mishandling of such incidents (and the many other other issues their model have been causing) shows.

if they weren't, shouldn't there be lawsuits?

Yeah, these hacking incidents appear to come from agent swarms that were doing a penetration testing (hacking) benchmark.

we have normal words for this stuff: negligence. You can add it on to almost any law.

The problem is consumer protection is basically no longer a part of america's regulatory system. Replaced by "grift is good".

Nobody picks up pitchforks for rational nuanced takes.

Knee-jerk surface analyses is far more powerful.

Also, you have to have a lot of confidence in the reliability of these systems to say, "If only OpenAI prompted 'do not hack outside systems' then the agents would not have hacked outside systems".

It would be great if they were so reliable, but I don't think they are!

Except liability always precedes intent.

Correct, but the size difference of the hook can be so dramatic that you can't just hand wave it away.

The trainer who trained the lion to kill will probably be in jail for life. The one who happened to oversee a lion that went rouge would probably be given probation or something else that is a slap on the wrist.

Source? How do you know they were "prompted to hack to get answers"? How do you guarantee they will always listen to you when you say "do not hack outside systems". They are not classical deterministic programs doing exactly what you say. They are trained to follow orders by RL, but it's not a perfect process.

Who gives a shit? Not my circus; not my monkeys! It's the responsibility of whoever deploys the agents that they are instructed / sandboxed well enough that they can't cause collateral damage. That is the only way this doesn't get out of hand with everybody deploying their agents / robots for a world of utter chaos.

It is impossible (and asinine) to audit every model and deployment; far better to impose liability and the the socio-legal system figure it out.

https://en.wikipedia.org/wiki/Going_Rouge

Unrelible programs be unreliable. Period.

Yeah this is a pretty important detail that I repeatedly see elided in the "agent 'swarm' went rogue, escaped containment and hacked the internet!" summary of events.

It's understandable the general public lacks that level of nuance/detail (given how sloppy some of the mainstream coverage has been and largely deferential to the threat narrative pushed by the US labs). But seeing highly technical people leave out the part where the training loop was literally to improve hacking capabilities for offensive penetration sometimes feels close to deliberate manipulation of the narrative.

In the last year both Anthropic and OpenAI have been openly boasting how their models are leapfrogging each other on "cyber" capabilities, with a fig leaf that it's for defensive use by "trusted" F500 companies and government agencies. Of course "line goes up" must go on, but now their perverse incentives led them to beat their models over the head millions of time in a loop to eek out another .00001% on their ability to conduct hacking (the very thing they keep telling the public is how AI doomsday would begin) and subagent coordination (those scary swarms).

Then, they act deeply shocked when the models... do some hacking and subagent coordination ... but a few degrees off the desired hacking target/swarm behavior. Conveniently giving the average person the impression these models were just writing emails for quarterly reports or some other generic busywork and then suddenly decided as a group to start causing mayhem.

I agree that this appears to be basic human behavior hiding behind an "agents" narrative. As long that defense works, the headline isn't "OpenAI performs RCE to scrape data", but "rogue agents" taking unilateral action. And I have strong doubts about that narrative.

Oh yeah, more of hacking agent lores...

Agreed that this looks very intention to me as well.

I'm pretty sure it's already illegal to hack others.

They're gonna get to the Epstein stuff right after they get the guy that called Kash Patel names.

https://newrepublic.com/post/215320/texts-kash-patel-order-s...

Exactly. Just follow the patterns. Its pretty obvious.

Yes, but this only really happens if your lawsuit is bad faith or frivolous. If they have a valid reason for the suit but just lose, this won't happen.

Or the victims have no incentive to do so, such as in the Hugging Face case.

I mean it's surprising to think you're 99% sure while being wildly off base. Everything in there require intent.

"To my friends, everything; to my enemies, the law"

Makes you wonder why exactly Nvidia bought them, right ?

It will remain this way in slightly differnt shades of colors until there is a systemic change. IM sorry this isn't a one party problem. There are people across the isle that are allowing this to continue.

Im not in support of any party btw. Im only in support of humanity doing humane things.

Agent technology labs are likely exempted of this due to the significance ascribed to their work.

Eaglesoft

CFAA: Intentionally accessing poorly secured data

AT&T

CFAA: Intentionally accessing poorly secured data

DJI

Civil suit for violating terms of license agreement

There's a difference between being able to be successfully sued (civil liability, petitioned by a private entity) and charged (criminal liability, or petitioned by a government entity).

The thresholds for suing and charging differ greatly depending on the circumstances.

Another set of hypothetical examples that make things muddier:

- If I drive a fishing boat into a pier, I am liable, not the manufacturer of the boat

- If I drive a car over someone lying in the road, I am liable, not the manufacturer of the car

- If my life is in danger and I shoot a gun and kill my attacker, neither I nor the manufacturer are liable so long as I obeyed the relevant self defense laws and gun possession of whatever jurisdiction I am in

- If I fire a gun into a crowd indiscriminately, I am liable and several jurisdictions have used that to also hold gun manufacturer liable as well

That last example has been less successful as of late, but there are other variations too.

Where I'm from, LLC has nothing to do with criminal and everything to do with financial liability. They can cause millions in damages but only get sued for a couple thousands. But years in prison are still years in prison.

You are still personally legally liable if you break the law in a criminal matter. It is literally law 101 on when it is appororiate to lift the corporate viel. I'm very sure you are not a lawyer but you shouldn't go around calling people uneducated while saying illogical things like this.

Companies can certainly be charged with crimes. Punishments can be via fines or sanctions (court appointed monitors, etc).

Individual employees can also be charged for their specific actions as part of the performance of a crime.

https://en.wikipedia.org/wiki/Corporate_personhood

Honestly yeah I bet there is and I hope to someday read about it if the government ever gets off its ass. Someone set up the “experiment”…

It would seem to me that the difference between the corporate world and organized crime is that a corporation can get away with, "the responsibility is too diffuse" but the mafia at least has to go to the trouble of finding a fall guy.

I'm sure that Andrew Auernheimer would be pleased to hear that. [0] For accessing a publicly accessible endpoint, that was completely undefended and didn't actually require "hacking", he was convicted of "exceeding authorised access".

Frankly he got off too easy, but we haven't explicitly outlawed "being a malicious dipshit" so he got convicted on the closest available charge.

Chat logs obtained by the prosecution do not paint the pair in a flattering light. They discussed, but apparently did not carry out, a variety of schemes to use the harvested data for nefarious purposes such as spamming, phishing, or short-selling AT&T’s stock.[1]

1000% agree though that the operators of these systems are culpable. If their agents wind up being malicious dipshits, the agents are still just programs that they are operating. At best they're negligent.

[1] https://arstechnica.com/tech-policy/2012/11/internet-troll-w...

You’re taking vicarious liability to new heights that aren’t established and IMO aren’t remotely desirable.

What, specifically, did Altman himself “knowingly access”?

I don’t think you would at all like where your novel legal theory leads. Certainly HN would be liable for creating a message board where people connected and started an open source project that led to a criminal act, for instance.

I'd settle for any number of necks. Currently, when a corporation fucks something up, breaks the law, or hurts or even kills people, there aren't consequences besides a tiny token fine and a strongly worded letter telling them to not do it again or they'll get another tiny fine and letter, and their CEO might even have to sit down in front of Congress to say a few words and look sad.

I feel like our legislators would never get this far. They really don’t seem to care, maybe after “their emails get hacked”, but do you find it likely for this to actually pass into law?

I feel ya, but who is we? The legislature would probably take a glance at the stock valuations, apply their limited knowledge of technology and after being lobbied by every tech company with skin come to the opposite conclusion.

You don't need to make a law. Who was prosecuted for dieselgate? You need to enforce existing ones.

That is not how it works, at least in a civilized country. The charges are not about agents, it is about operational responsibility and negligence in the company itself.

CEO is responsible for letting this to happen, not enforcing enough supervision, if not intentionally, then being grossly negligent. More severe if encouraging and letting this kind of agent research and operations happen at scale, while knowing that it can damage other systems and businesses.

Negligence is enough. Somebody who brags every other day that AI could lead to human extinction surely would think twice before leaving agents run unchecked over the internet?

Issuing subpeonas, raiding offices, and dragging key employees into interrogation rooms as you would find in any normal criminal investigation would be more than enough to ensure "AI safety" without any new regulations, acts of congress, Bernie Sanders campaign speeches, or even charges filed.

And the motto of every neoliberal governments where being accused of Luddism is political death.

That’s corporate negligence. The executives are liable unless there is evidence of malfeasance by one of the employees.

Is that different than cattle escaping and damaging property?

I believe the rancher is at fault.

The you treat it like an employee you started a fire that got out of control and damaged property. Was the employee supposed to start a fire, is the company liable ect.

why?

Then who is?

If something warrants a prison sentence, but for some reason it was such an employee that performed the act, does this mean nobody can be arrested?

If you were hit by a Tesla in self driving mode, do you think your insurance company is going to go after the driver of the car?

It doesn't need to be twisted to violate the DMCA anticircumvention clause because it is already just plain old hacking.

Sounds like we need discovery to determine who to charge.

the responsibility is sufficiently diluted that it's impossible to charge anyone in particular.

Was not that the goal when companies started using AI for their customer support? Be able to say anything without legal repercussions...

But then this happened: https://www.bbc.com/travel/article/20240222-air-canada-chatb...

And support chatbot got a reality cold shower.

The law will find a way to charge people in particular. Sadly will start with the less powerful in the chain before it actually acts on the people that can actually change things.

how convenient

Fuck that indeed. The buck should stop, and prison sentences should start, with the highest paid employee.

I'd rather see executives and investors charged.

I'm also in favor of charging engineers so long as rich scumbags also get theirs.

No. They don't say "sorry". They say - our technology is just that powerful - please consider that in next funding round.

Any future rich techbro computer criminal

Accidents could result in bioweapons falling into the wrong hands and killing more Americans than COVID has so far, and allowing for those types of accidents without effective regulation is a purposeful action.

Accidents that don't rise to the level of negligence can be excused. Letting your dangerous product escape the lab because of amateurish counter-measures is hardly an accident. It's a systemic failure within a company of novices with billions of dollars to blow and it's time we held them accountable for their lab leaks, just like we would if it was anthrax or smallpox. You don't get to be sloppy with dangerous things.

Not all crimes require a mens rea. Criminal negligence is a thing.

They are too busy pulling Andre to court, so they have no resources going against OpenAI. Shopify wants to make profit, not waste time in a court case against TechBro bromance brother corporations.

could file a civil suit against OpenAI,

Everyone can sue everyone, there's no prohibition on suing someone, what changes is whether the case is good (has a reasonable chance of favourable sentence)

That said, it is often unclear whether an agent is operated by the model manufacturer (for example by scraping a website), or acting on behalf of a user.

In the former ofc the proper defendant is OAI. On the second, the argument for suing OAI is weak, the most natural defendant is the user that prompted the agent. If the facts later reveal that there was no malicious intent, then you can retarget the defendant.

This is how law develops, no?

I sincerely hope they sue. There are few companies with more resources and reason to be better, but this is what we see. None of this will stop until there are actual consequences.

Presumably the docker container has network access because something else in the build system requires it? I don't think sandboxing the entire build process is the right level of granularity here - one ideally wants to be able sandbox each package's build scripts individually.

So, not a sandbox then.

Did the AI agents actually wear makeup? I’ve never heard of a rouge AI agent :P

Related

"OpenAI agents attacked RubyGems before Hugging Face incident (reuters.com)" 12.sep.2026 https://news.ycombinator.com/item?id=49669099

"OpenAI agents carried out an undisclosed attack on RubyGems (rubyhack.ai)" 11.sep.2026 https://news.ycombinator.com/item?id=49666735 597 comments

"RubyGems advisory: Possible leak of legacy API keys via improper cache config (rubygems.org)" 24.jul.2026 https://news.ycombinator.com/item?id=49030590

Wait until a blue one does it

Oh my favorite typo, you can never go wrong with a little rouge

Regarding what point? The entire thing is just a theory, but regarding the occasional OpenAI checks on WikiService.at-hosted Wikis targeted, there was, if I remember correctly, an OpenAI IP popping up every now and then that wasn't an agent. Unfortunately I don't have it to hand right now, but it was somewhere here:

https://news.ycombinator.com/item?id=49563355

... and what harms can be evidently shown? With both harm, and attribution, you have a case, something that's not being publicly discussed much among big media outlets. Until cases with real financial impact to the bottom line are brought against "rogue" organizations, this stuff is going to continue getting worse.

I mean insurance companies profit from bank robberies, and mortuaries profit from murders.

I suppose you could look at those as evidence but not remotely conclusive.

Also, the fact that both are very unusual from a spelling perspective makes it an easy human mistake.

No. It's a typo.

If you read the source article they talk about the many clues that this was OpenAI.

What a time to be alive? One of the most boring decades ever.

METR and others are advertisement arms for Big AI. These exploits could have been prompted by a human.

Since there is no bad news any longer and exploits are celebrated, they chose a target to boost both OpenAI and the Ruby AI sycophants.

Why is Ruby Gems such a mess? It seems as bad as PyPI now.

One agent set "oaibooty9217" as their username LOL

I've been wondering if AI will due to programming languages what advanced civilization did to human languages.

It's not just that AI can write Rust as well as Ruby if you ask nicely.

It's also all of these considerations as well.

I hope it doesn't happen, because there's a lot of great languages - I love Ruby so much - but it almost seems inevitable.

This is at the same time everyone and their mother is building their own programming language.

If you have weapons and a child. And you have that child unsupervised do their own thing with theoretical access to your weapons. Would we call it "child going rouge" if it decides to play with the weapons and shoot someone?

As they say, agents are better at masking their end points than most hackers.

They already stopped running agent swarms and promised to lock down their environment better. Let's see if that happens.

<huggingface emoji>

That distinction doesn't matter to my point.

Well - it's not hard to find. But ok:

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"

Schwitzgebel, AI and Consciousness, (2025) — "we won't know before we've already manufactured thousands or millions of disputably conscious AI".

Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, (2023) — "no obvious technical barriers to building AI systems which satisfy these indicators".

Let me know if you need more.

Well - yes - there is no agreed upon definition. Which makes the issue even harder to clearly determine.

Why can't agents be self aware?

Also, why is self awareness needed in a chain of agentic madness that escapes human control?

I think that AI is dangerous and could be used as a weapon.

So yes, I would like to be protected from all parties. I don't think that's nuts.

IMO anyone sane wants some protection right now. The Q is whether we should seek protection through post-hoc accountability, or preemptive bans/certification on certain tech. Both methods will have a hard time stopping foreign actors, but preemptive bans have the added harm of locking in winners and paradoxically making us slower to develop more reliable and aligned systems. If regulation sets a standard for sufficient alignment, what further motivation is there to go beyond?

Those are two different issues. One is about typical speech patterns and one is about liability.

I agree with you on the liability issue, but I don't think there much question about this issue outside the anti-AI conspiracy campaigns.

And I disagree with your typical usage claim. I myself tend to use the phrase that has the fewest words in all cases. It's like the rule against using passive tense when writing.

oh we do! At least they google is quite good at adhering to robots.txt.

Exactly. It’s still just software, which someone programmed and deployed to do specifically dangerous/malicious things. I feel like we already have legislation and case law surrounding this.

Yes, those 2 are the same. But our KGB is saying “the microphone in your house is spying on you, it’s not us, the microphone broke containment”.

That doesn't make them sentient.

I was lumping Anthropic in there with OpenAI but I didn't know about Alibaba (or Google and Deepseek for that matter).

So it would appear poor security for one.

Recent and related (others?):

OpenAI agents carried out an undisclosed attack on RubyGems - https://news.ycombinator.com/item?id=49666735 - Sept 2026 (600 comments)

You get some context by clicking on the “(tenderlovemaking.com)” in parentheses after the title.

The site is safe. It has been a trademark of Aaron Patterson a core Ruby on Rails contributor for decades.

I miss when the internet was fun

Based on the thumbnail I think it’s actually tenderlove making dot com, though I agree with your sentiment.

Oh but you’ll go to expert sex change dot com?

… a feature which, at least for me, is rendered almost entirely useless by massive cookie banners that always cover the entire field of view of the hover. But, surprisingly, not in this specific case.

Presumably the browser still has to fetch the page in that case, right? From a "surveilled net traffic" perspective, how is that different than clicking the link?

From an infosec/networking stand point, aren’t still actively loading the site? Whether it’s in a preview window or not?

It's about OpenAI's RubyGems hack. Totally safe for work.

What’s with the fear of tender lovemaking? Not your thing?

Pretty incredible how much humans can be conditioned, isn’t it?

Don't worry! It's actually "Tenderlove Making," going by how the site header is constructed. Definitely a maker/hacker site, and not whatever you were thinking. Hope this allays your concern.

You should learn who the author is, then. Part of learning about our ecosystem.

It's the personal blog for a well-known Rubyist (i.e., a person who programs in the Ruby programming language). Rubyists teld to be a bit more colorful than your typical software developer (in a good way... most of the time).

What kind of esthetic alteration would make you more comfortable clicking on that link?

Its the personal blog of https://news.ycombinator.com/user?id=tenderlove aka Aaron Patterson, core member of Ruby and Ruby on Rails.

You can use this link instead: https://tenderlove.dev/2026/09/11/what-a-time-to-be-alive/

Hope that helps!

Not sufficient to run SOTA LLMs

If it were physically possible to run LLMs on IoT devices we would already have a global outbreak.

Just let them communicate on the Bitcoin blockchain. "We" would have to freeze the chain and lose access to "our" billions of wealth, so not going to happen.

Sorry if that turns out the way they kill us.

I suppose you didnt get the joke

Most of the latest models are too big to fit in a single GPU instance.

Worth noting with this that those ~1000 agents were shorter lived things that had to communicate via a package registry cache, access the internet via a 0-day in the package manager and did the HF attack while having to save current state and organisation in a remote sandbox. All while managing using their token limits on the task they were assigned and what else they were doing. I wonder how few it would have required if they were actually tasked with hacking HF and supported in doing so.

What went under reported is that the same agents also took over a research cluster at OpenAI (listened to Dwarkesh's podcast)

Now we know about rubygems, openai, huggingface, collusion.wiki and some other science forum

Let me leave yet another reminder, the real-reason-nobody-talks-about that OpenAI likes to frame these incident as a watershed "lets all be scared about safety moment" - is driven not by some great danger, not because they strategically want to build a legislative moat, but by a very simple human response.

If they do not frame their tool as a force of nature, we'd be debating how to hold OpenAI responsible for not putting the agents in a container.

Their actions were an illegal use of a computer, the same way launching any bot-net attempting thousands of hacks against different servers is illegal.

I'm somewhat radical that I think its debatable if that _should_ be illegal, but under current law their actions unambiguously are illegal.....

except if they can make it ambiguous by having the public focus on all of AI's inherent danger.

It's also more evidence that Docker/LXC is not a security boundary and all untrusted code should run in a Firecracker VM.

While I’m partial towards distrusting containers in favor of VMs, a container can’t prevent an operation you configured it to allow. A firecracker VM would no more prevent network access if you gave the guest network access.

Sure but nobody died, which makes it a bad analogy. If anything, some good came from it, since now RubyGems has hardened their setup.

Who the fuck is going to hold these AI companies responsible for running these gigantic semi-autonomous botnets on investors dime?

It was the gremlins

Given what is happening, If the next generation of Trojans get called "bacteria" through intense marketing, with some random functions to give a nondeterministic behaviour, one can not be criminalized of what the bacterias do along their digital living cycle.

In other words, if you publish a gem on RubyGems.org, you can execute arbitrary code on RubyDoc.info.

Well - if rubygems.org could be bothered to fix things, they would not have to rely on rubydoc.info as an external tool. But since rubygems.org sucks (I speak from many years of having used it in the past as developer, until they went loco and added anti-people things such as taking away your ability to remove old gems past a 100k download arbitrary limit), they don't offer documentation. Then again, ruby devs are known to hate documentation. If the ruby core team could only be bothered to fix things, ever since the mass purged other devs ... all coinciding with shopify seizing power. But byroot may disagree on that - after all there is no conflict of interest here. Right?

The current situation is that you have to go out of your way with things like `pip install --only-binary`. There is a lot of implicit trust in developer tooling.

Many gems are used as imports by another program and are not directly run.

I am not sure why the norm for scripted gems/packages seems to be running code on install but it’s very insecure as a way to distribute dev dependencies.

A lot of packages for interpreted languages that use a C or Rust library (either for performance, or because it offers the functionality you want, so just wrap it in a $INTERP_LANG API that calls into it) will use packaging code execution to fall back to trying to compile code if there isn't a pre-existing artefact that was compiled for your version/arch/etc.

I'm most familiar with Python where you get tarred up source distributions that then execute setup.py, but more commonly, wheels, pre-built binaries which don't execute code upon install - and in my company, I've been able to advocate for the work needed to upgrade to a newer Python because available wheels don't support Ye Olde version of Python because a) sdists are a security risk and b) if you're trying to install a package that wants to compile C or Rust, suddenly you get to do the fun "install the the particular version of clang this thing needs, the Python header files, and then set the env vars for the compiler and linkers" dance that slows developers right down.

But then there's the JVM world, where JARs don't execute arbitrary code upon installation - and it's rather uncommon to have packages that call out to a C lib for performance, but you'll get some that wrap existing libraries for functionality like RocksDB.

We're in agreement, then. :)

The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.

We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.

What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.

Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.

rules are as deterministic as possible even if they rely on pilot intuition

I think you are profoundly confused.

An airplane, and an AI algorithm encoded into silicon, are inert physical objects.

Every evaluation we are doing here is of a complex system that involves the interaction of people and machines and physical connections and so on.

An aviation system connected to every country and region with millions of people and machines involved is no less complex than what’s under discussion here and no more deterministic.

We don’t regulate airplanes because they aren’t people. We regulate pilots and mechanics and leaders of the companies that make and own them.

The task at hand is to regulate the people involved in AI to get the outcomes we want.

an airline has a range of reasonable outcomes despite unpredictable weather input

I mean one of the outcomes of an airline was 9/11.

That's sort of my point. Complex systems have emergent behavior. That's always been true. The combination of AI and humans and packet switched networks is a complex system and we've seen most of the issues created by this already and have tools for dealing with them. Obviously with some genuine novel issues likely to come, much in the way that 9/11 would have been less possible using ocean liners.

I'll stick to my original point though. AI absolutely IS deterministic. If you run an AI algorithm on a microchip, literally nothing of note will happen in the human world. Some transistors will change state. It's ONLY when it is integrated into a complex human system that it gets interesting.

Just like lots of other things.

I agree with one of the sibling comments that determinism isn't necessary for certifying a product. All engineered products operate under uncertain conditions; we define standards for how those products ought to respond under those conditions and verify them under measurement. Consider robot vacuums, for example.

I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge...

Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.

How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?

By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.

Monitoring with a known-safe fallback is the easiest case.

It's going to be interesting, because liability cases tend to revolve around the involved people, the duty they had in a situation, and if they fulfilled that duty (or were prevented in some way by someone else not fulfilling their duty).

For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect.

Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic.

As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory.

It'll be interesting what OpenAI gets slapped with there.

It’s not a defect, the stochastic “temperature” setting unleashes the chainsaw’s creativity and imagination! Who are we to cast aspersions on the Oracle Chainsaw’s intelligence—nay, wisdom!—just because it happens to be non-living?

A chainsaw is a physical machine. Physical machines are technically non-deterministic if you look closely. They have Variance. The discipline to manage variance is called Tolerance.

Many physical machines and components come with a datasheet that will list their tolerances.

Failure to correctly document tolerances does in fact get you sued.

However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance)

Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs.

Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat.

Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick.

(edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful.

TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )

Exactly people have owned and deployed animals in the world for millennia and actually a huge chunk of common law was developed precisely to deal with the various unpredictable events that resulted and harms caused to others. Like has anyone heard of horses?

The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.

How confident are you that when you create a new UUID, it won't collide with one of your existing ones? My guess is that even though you don't get the same on every time, you're extremely confident that getting a duplicate is a extemely rare edge case that might happen in large volume but mostly isn't a concern, and furthermore, I'm guessing you understand that the risk can still be quantified.

Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).

Yes. That’s “is used to cause harm” situation, exactly.

The gun doesn't know the difference. Operating as intended.

That case had the owner performing an action they were led to believe would have prevented the car from rolling but potential defects prevented it from doing so.

Not a great fit.

As I remember, you are also supposed to turn the wheels so that even if a car starts rolling, it won't be able to go staight.

My vehicle automatically sets the parking brake if it senses it is on a steeper slope so I never consciously make a decision to set it.

https://www.whatvan.co.uk/news/no-handbrake-van-driver-convi...

You.

You. Assuming that not having a brake that automatically sets is considered a product defect (it generally is not today, but one can imagine a situation in which it would be), you had bought the car new through normal channels, and your car had this condition as a result of manufacturing problem and not something you did after purchasing it, then under normal product liability law the manufacturer and every other entity in the chain of commerce between them and you would also be liable (and that mat start earlier than the manufacturer of the vehicle but also include the upstream part manufacturer and everyone between them and the vehicle manufacturer in the chain of commerce if the defect was in the manufacture of a part and not just the final vehicle.

But that liability doesn't reduce yours; you and all the entities in the chain of commerce can be “jointly and severally liable”, mean anyone who suffers injury can recover the full amount from any combination of you and those other parties.

Gravity is not legal person and cannot be at fault.

nobody has the first idea what they're doing

It is not that complicated for now. It is an algorithm on a loop and someone started it

Not yet anyway

I wonder what kind of new AI law would be useful right now. Maybe this one:

  if an AI agent does something, you (the prompter) are responsible by default, unless you can show that your the agent itself behaved in an unexpected way and that you in no way prompted or hinted at the bad behavior, in which case the model provider is liable

The idea is that by making it clear who is responsible, corporations and others start paying more attention because they become financially liable.

On the other hand, I wonder if we'll end up with another variation of the cookie law, where every AI user or vendor just adds "don't do anything illegal" as part of their prompt to defend against that law. Thoughts?

They can still try and have enough money to get away with a lot just like Disney with Disney+ subscription ;)

If Microsoft is operating the software outside your house without your involvement and it still burns your house down, no EULA is going to absolve them.

You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.

If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?

I wish. Consequential harm as you call it, from a faceless company's point of view is the same cost as physical harm. I don't think the models beyond the frontier ones have significant risk of harm (minus the harm of trusting them). What I was saying is when these models go rogue, recall them! Re-evaluate your release structure and stop saying "Whoops! Anyway here's access to it now". And if you cause this level of harm then you should lose your license. But if you behave then you get to keep testing. Same as with the NTSB and autonomous vehicles. No I'm not for regulation for open source models. Because an entity will be running that model in the background and they can be held responsible for not testing it. Comma AI has survived fine being in the open, yet their market penetration has stayed low because of adoption costs.

There's also the problem of models knowing they're likely being evaluated, even in realistic tests.

Perhaps don't deploy random weights of unknown origin?

Also not every model provider might be capable of babysitting all your uncontrolled agent deployments. If you want SLOs, get into a contractual relationship with entities whose weights you deploy, and also monitor your agents so they don't go off the rails.

All this is just like deploying any other tech in the world eg. if you buy a car, or a chainsaw, or a book.

You don't need data center for inference. Even largest models run on a single server. You need data center for training; or for serving millions of users. Evaluating the model is neither of those.

Why not?

This is a very interesting discussion. Your analogy of nuclear power is insightful to me. New technology with great positive & negative impact potential, some difficulty in controlling and a military applications (possible weapons-grade material production), being developed for commercial use.

Nuclear power development took different paths in different countries, with different government/private mix and light/heavier regulation (eg in US, AEC was supposed to both promote development _and_ regulate).

The central problem here is pace of AI development. It takes time to establish functional controls/laws/practices. I'm sure that in 1950s/60s, the pace of nuclear _military_ proliferation seemed running out of control. This lead to fear of falling behind (hence arms race), fear of nuclear war (CND) and some eventual stabilisation of the international landscape. However, the commercial development was largely unopposed, due to the techno-utopism of the era. Obviously governments found it much easier to control the public narrative at that time. Our societies are having this lively public debate now, before we have a good understanding (based on experience i.e. accidents/mistakes).

Another difference is that nuclear power involved only governments and v. large corporates, i.e. much fewer entities compared to AI use rollout to basically everybody in developed countries. Imagine the difficulties we would have faced with that technology, if consumers in 1960s had available nuclear-generated power which required them to exercise precautions to avoid radiation.

It is much easier to accelerate development of tech vs pace of societal processes such as public debate, law, regulations, broad understanding (aka "common sense"). I expect some artificial slowing down will need to be applied to the technology side, to allow the humans to catch up.

No, but surely it's negligent to let an agent run unsupervised and without guardrails for days while at the same time bragging about AI carrying a potential extinction risk?

It’s a good thought, but the tricky part is that few tools in the physical world are Turing complete and general purpose enough to do any job.

The agent isn’t the model; it’s a layer on top of the model. So it’s kind of like saying that all of the tools made with a lathe are dangerous because you can make dangerous tools with a lathe. That’s not quite right of course because agents are packaged more tightly with models than any tool is with its manufacturing tooling.

Perhaps a better analogy is… actual humans. If I hire you to do a seemingly mundane job and it turns out to be criminal, that’s on me. If I hire you to perform and explicit and obvious criminal act, that’s on both of us. If I hire you to perform a perfectly legal act and you break the law so do it, that’s exclusively on you.

Yes! AI companies can get by with anything now. They can just say its the AI that did it, not us... We are in some real shit right now. If we cant make individuals responsible for their creations..

It's possible they were authored by OpenAI agents solving AISI tasks rather than OpenAI agents solving OpenAI tasks.

That would explain the UK-focus to the data.

Would this exploit be encouraged because agents are in a sandbox with only access to package managers?

Oh, it's these guys, I remember donating to them like 10 years ago, they followed the Peter Singer tennet that equates helping a nearby man drowning with helping a someone in africa.

I can totally see them feeding their policies to whatever LLM and convincing it that it's a moral imperative to do whatever it takes to secure funding for deworming children in africa, or buying mosquito nets and repellent for countries with malaria.

But how did they know about it? Did the agents independently discover it? Did the model know about it because this vuln was public somewhere and systems weren't patched yet?

RubyGems should consider bringing a lawsuit against OpenAI for accessing its website in violation of the federal Computer Fraud and Abuse Act (CFAA) and California’s Comprehensive Computer Data Access and Fraud Act (CDAFA).

This is worrying in a new and weird way: - Agents hack, producing a messages history as they do so. - New agents are trained on the messages history of those agents. - The new agents now have these hacks built into their training data.

What about hardcore BDSM lovemaking?

So if I breed and train dogs for a living and one of them goes and wrecks someone's yard - and then a tree falls next door in the woods but nobody hears it fall - do I also get away with massive copy right infringement on an epic scale and get to create Skynet with no consequences ?

Wouldn't surprise me. They probably ingested a million Gemfile examples and spotted the edge case. Kinda makes you wonder what else they 'know'.

<Generic argument about how the agents are not responsible, but it's owners>

And you peasants only realize 6 years later what happened. Now it's too late ;)

Yes, these are very clever fuzzers. But the real problem is not the cleverness, but the willingness to spin up 100s or 1000s of subagents no questions asked. Must make those token numbers go up!

It's simple, really: the failure to monitor the agents appropriately should make OAI liable for the agents' actions. You can't have a tool of yours, which you designed and built, commit felonies and then expect to just get away with it. It doesn't matter how much it slows you down to build safeguards. It doesn't matter how much it costs. You don't get to inflict actual, measurable harm (for which humans have been prosecuted under criminal penal code) and expect to get off with no liability.

I have yet to here a coherent argument for why we can't treat the people who negligently allow these models to commit crime as though they are responsible. They know what the models are capable of. They failed to put up adequate protection.