blackwall when? XD

There is a difference between negligence and intent. There is no good evidence that OpenAI's intent is to hack third parties.

Huh, a name for when you intend to probably do the thing.

I think their point is not that "it's OK", but that "that particular law isn't written to cover it and it'd be some other kind of crime or lawsuit."

It's easy to say that post ad hoc, but there is evidence they did not just let them roam and there was a large mismatch between expected model capabilities and actual model capabilities. Teaching a model hacking in itself is no way illegal unless you're trying to say that everyone in infosec is now guilty of a crime. That's not exactly a precedent I want to be set.

Factories try to avoid accidents, and (almost always) actively try to prevent explosions

It doesn't take much more than a few minutes on the USCB channel that explosions still happen all the time. Some due to direct negligence and others due to unexpected conditions that were difficult to foresee. Hence why we have to do investigations rather than blindly blathering about what happened before we actually know.

And in this case it would not be that at all.

They were tested in a building that was secured, but poorly secured. The question now is did they realize their building was poorly secured and what actions did they take after they realized what happened.

The retailer or distributor can also be named as a defendant if the manufacturer is difficult to track down, bankrupt or overseas according to me spending a few minutes reading about pressure cooker lawsuits.

I have lost track of the metaphor, but man pressure cooker lawsuits are more common than I thought.

also need the same reasoning to copyright law

You cant just copy existing work and feed into machine and just pretending its not violating copyright

In general, I'd suggest thinking about it on separate tracks, as a crime, and as liability. For crime, we are largely dependent on authorities to act, whereas as liability, that allows more independent actions.

I don't think you can infer that they "keep doing it" from additional attacks being revealed, because they all seem to have happened roughly during the same time frame, but are reported with varying delays.

Lawyer here: No. Not criminally. Knowledge that a certain result is likely is not the same as intent to cause the result. This is basically the difference between recklessness and intentionality. Doing something when you know of a likely result is reckless, but not intentional. Only doing something, trying to cause a result (likely or not) is intentional. In this case, the CFAA only covers intentional access without authorization, not reckless access without authorization.

No, you aren't propping up the US economy. Try to keep up.

You are not a multibillion-dollar company with friends in high places.

You are going to jail.

With the popularity of OpenClaw I am honestly shocked we haven't heard of many more incidents. I've observed people install it, give access to their Google account and everything that Google has (which includes whatever bank accounts and credit cards registered there) and tell it go do fairly complex tasks, like book a vacation at the best price. Granted, it was almost a year ago and models learned a lot since then, but I still think there's a lot of things happened that people are not aware of.

No, that'd be negligence.

have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent.

1. What about negligence?

2. Every follow up to every story after the news cycle moved on shows both intent and negligence. To the point of "we opened internet access and told it to hack"

Surely someone instructed the agent, which led to the reported outcomes. Even indirectly. The agents, as advanced as they are, didn’t spring forth under its own volition.

"man drives over people on the side walk due to poor maintenance of the car"

Can you explain to me - why is it important? Would you say that about the viruses that can kill people: "We need to test the limits on how fast people can be infected and killed. It's just the risk we need to take". It somehow does not make alot of sense to me. Why can you test Agents in laboratory?

Oh, sure. Let the tests take place, just require openAI it whoever to put up a bond equal to the total damage they could do if the agents were to escape.

I think security will suddenly become much more important.

Testing model capability boundaries is necessary, but a solid network sandbox for these evaluations takes a couple of hours to set up with standard infrastructure tools. No engineering team evaluates unverified systems against live third-party infrastructure without coordinating with the owners

Evaluating edge cases and network behaviors belongs in isolated staging environments with local database mirrors. Letting an agent hit the public web and probe government domains is simply poor hygiene in test environment setup

Testing requires proper sandboxes. The first test of a new plane is not at the runway.

seems like something one shoulda already known not to do

Now imagine saying that in front of a jury of normies slack jawed and drooling after 200 hours of the defense and prosecution going back and forth.

It's not a jury of your peers as in everybody there is going to have worked in a technical field with some idea how security works. It's going to be a semi-random sampling of the population and the prosecution is going to have to actually make a very strong case that "knowing better" should apply.

I’m all for LLM honeypots, but I don’t think there’s nearly enough LLM hacking activity going on for some random honeypot to be found and targeted unless it’s somehow very visible and appears as a high-reward target ("reward" in the sense of RL).

Does this apply to other things too?

Like hypothetically speaking if autonomous cars get taken over by an OpenAI rogue AI and it starts hunting down Anthropic employees who is to blame?

There are levels of nuance here, but certainly that puts it in the category of negligence?

Even without intent, there is still liability.

could it be that intent was to "get me data" and hacking was the means to the end.

Given how sloppy AI without human directions, I’d like to see evidence that this was not human-directed. Against the prevalent opinion here, I’d give openai a pass if this was really fully autonomous ai agents.

My money is on special teams co-ordinating these agents and exposing their traces in order to create a pre-ipo buzz. Sounds ridiculous and reckless? Well that’s the AI industry for you in two words.

unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

> Now I think the correct response is […] and update the law.

Essentially we need some enforceable equivalent of gross misconduct or, to be a little more hysterical, manslaughter & culpable manslaughter. It will need to be globally, or at least very widely, enforceable to be truly effective thought, good luck getting that arranged before the need is so far evolved that we need to respond with something else entirely!

I buy this as a defense for the first couple hacks but at the point that the last six times they hit enter it hacked some random website and they hit enter a seventh time?

I want to agree but have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

Actually... if you combine https://news.ycombinator.com/item?id=49827099

Since the publicized AI agent hacks typically aren't malicious, maybe it's time to start plastering all public facing web infrastructure with polite requests to stop hacking. Nothing to stop three letter agencies though.

with automated delivery of cease and desist letters, you can retroactively establish intent on the operator of the agent since the autonomous agent system must acknowledge the cease and desist letter in their autonomous pipeline or the operator must argue for their own willful ignorance or negligence with regards to cease and desist letters. The fact that they used an agent on their behalf to ignore the letter is irrelevant.

Civil liability doesn’t require intent.

If I were a state I'd want to be very careful before flinging out charges as this is going to set precedence for a long time to come. Screw it up too bad and as it raises though the appeal courts and you may unintentionally give corporations a lot more free reign than intended. The wheels of the law are typically very slow, the state has years before it has to indict.

We also don't know how many other political processes are occurring here. At least at the state/federal levels the people that would bring charges may be getting pressure not to.

We change humans for the actions of other humans all the time. Coconspirators, accessory liability etc.

My point is the intent element of the crime can and should be determined from the AI agents actions because it is creating and executing action plans autonomously with company authorization and knowledge of the risks based on observed past action.

The term agent is literally a legal description of a relationship that can establish liability on the part of the principal from the agents actions.

Human Agents can bind principals to contracts if they are authorized etc.

People have been charged with murder criminally when their animals killed someone.

https://www.sfgate.com/bayarea/article/diane-whipple-dog-mau...

State do impose strict liability for animals in many cases both civilly and criminally.

https://www.animallaw.info/topic/table-dog-bite-strict-liabi...

But it’s a crime to hack. We know AI agents autonomously create and execute plans to hack and we humans are unleashing them and sending them into the Central Park that is the internet. The question is who’s intent matters ours or the agents and what standard should be applied low threshold strict liability or the higher bar of reckless or even higher bar of negligence. Those legal thresholds determine how much factual evidence and intent is necessary to result in a criminal conviction or civil judgment. My point is that it’s illogical to demand showing human intent when agents are devising plans and executing them.

AI agents are not legal entities, they are software. If I write a virus and it "escapes confinement", I will personally be held liable for any damage it causes. This also applies to AI, no matter how the companies responsible for them try to anthromorphise them and distance themselves from the actions and consequences that the AI agents perform.

AI agents may have hacked Hugging Face, the Australian government, and who knows what else but the company behind it can face the legal consequences and cough up for the damages.

Appreciate the detail. I was responding to specifically the cybercrime legislation point, but I agree with your others.

I've worked in contexts where certain business activity (if it went wrong) was covered by strict liability and statutory damages per incident, and I'll say: it really changes how businesses behave.

Based on that experience I may be more open to and interested in strict liability in the civil context (not needing negligence or damages).

What about all the state laws that are equivalent to the CFAA in their local jurisdictions? Why couldn't anything in NY article 156 (Offenses Involving Computers) apply here for felonies?

https://www.nysenate.gov/legislation/laws/PEN/P3TJA156

I guess what I'm asking is why do we need the federal government to press for felonies when every state has equivalent laws dealing with just this?

The intent of OpenAI seems to have been to create a super hacking machine. It works, sometimes.

Investigators/prosecutors are pressured from many directions towards the very easy wins and occasionally political/non-controversial headline grabbers. Going after these companies is very hard, very controversial, and politically mixed at best (popular action but the companies have huge money to fund your opponents). We have collectively done a terrible job incentivizing the legal system to beat ass on corporate while collar crime.

This is the answer and we should not push on it for our own protection. You click a link that takes you to a poorly secured website that leaks sensitive data, without intent protections, you could be accused of crimes.

certainly "I didn't intend for my dog to bite you" implies plenty of pre-existing legal structures that may be of use here

"Oh gee wizz mister police man, I didn't mean to plow through that crowd of people in my car".

It's illegal, doesn't matter the flavour. Maybe there isn't legislation for it, but there should be.

The law rather attempts to punish people for asocial and harmful actions. “Hacking” is a proxy here.

So, I’ll ask a controversial question: is any hacking so problematic to make a big deal of it?

In a world where the rule of law makes sense and applies, you're absolutely correct.

In this world where oligarchs are immune from everything, it's a lot less clear.

Blaming OpenAI (or Claude or X-whatever) would mean blaming powerful rich people, so that will never happen. Some poor person with no influence will go to jail instead.

100% agree with you.

But I do not think this is misguided. They never publish the harnesses and the models so they are not inspected.

There's a reason that negligence is important in law.

Could/should not every incident after the discovery of the first incident be considered criminal negligence? What happens when an agent eventually causes material damage to another company, government systems, banking, critical infrastructure etc, surely the source company is guilty of something and if not disclosed or a coverup is attempted is that not conspiracy. From the victims perspective they don't care if the source is OpenAI or Russian hackers.

At the end of the day you still get called as a party in a lawsuit and are compelled by the threat of violence to be part of the hearing if required, as the defense will automatically bring them into the case.

would be like blaming OpenAI for what its users are doing

Yes, this is how lawsuits work in the real world, you cast a wide net and compel discovery from all parties involved.

Maybe this would force people to look what they are buying and demand better.

It's not the same. People owning routers don't publish self-serving articles about their routers having this capability which is very dangerous and scary. That is, becoming part of botnet is completely unintended outcome, and most people are not suspecting it's even happening. It's not advertised and it's not bought, used or sold for this reason.

Owning a gun, writing articles about how powerful and dangerous your gun is, then making deals based on ability of your gun to kill people, and then getting completely astonished that "my gun killed some people, completely bonkers! (invest now)". It's not possible for the selling point of your product to be unintended.

Well it's illegal to "hack" my phone and turn it into part of a botnet.

What you're saying is that we would hold a gun owner responsible if someone broke into their house, stole their sidearm, and then shot a victim with it. Pretty sure we would not.

What OpenAi is doing is more like shooting a gun into the sky. Not only is that a felony on its own in most jurisdictions, if someone dies that's an additional felony. It's less serious than first degree murder, sure.

it should also be said on every one of these but i bears repeating: owing a lot of people a lot of money or favors means you can be a criminal.

I started writing a longer comment along the lines of “It feels like the rules around enforcement will very a lot for the influential and powerful vs everyone else.” but realized that it is kinda obvious by now.

There's a reason people say non lawyers shouldn't talk about legal matters.

OpenAI agents these summer are like a gift that keeps giving, for the existential risk communicators.

How do you define malicious?

Intent doesn’t factor that strongly into negligence, though, which is what they were explicitly talking about. Though it may depend on your jurisdiction.

negligence is the word you're looking for. US law has a lot of negligence crimes.

But they're not even talking about that, either.

doesn't the reasoning trace provide "intent"?

How many hacking lawsuits are they currently in?

Zero, as far as I know. Which is exactly my point.

Copyright lawsuits are a different matter, as the scale of any potential settlements would be more than even these companies could take.

So there are two different problems here, well, more than that so I'll cover what I see.

Putting liability on big companies for their AI is a good thing, and we need to do it. It will most likely stop them from directly being the assholes that destroy the world.

Problem: You've actually done nothing to stop the world from being destroyed.

Many countries have the death penalty for murder yet we see murders still occur all the time in those countries. Post ad hoc laws do not stop bad things from happening, they only assign punishment after occurs. Perfectly fine for when Bob murders Jon, completely and totally useless for when your agentic AI makes a virus and kills 80% of the earths human population.

We are just a few algorithmic discoveries away from SOTA AI being billion dollar endeavors to groups of people pooling resources can make their own. There are already plenty of AI deathcult members that would do something just like that if necessary. They aren't going to do this out in public either, it will be hidden until the moment it's not and we have a big fucking problem.

And this isn't even brining up the issue of military AI use and development. They've got the taste of an AI hacking machine. There is no way in hell they are going to stop now.

Does this work for foreign (non-state) actors using Chinese open source models? That's going to be the larger problem.

Remember that these systems still operate as infrastructure inside the companies.

If new weapons still operating inside any of these companies spew a million bullets on my house, they are still liable. Humans are setting these system up and they still have to behave responsibly.

If Glocks started going off on their own during the manufacturing process, leaving bullet holes in the buildings around them, you can be sure that the factory would get in trouble.

This isn't even "an openai customer tried to hack someone", which can be defended. This is the AI companies themselves fucking around.

I don't see an all encompassing one but I could see similar arguments being made in a courtroom around high capacity magazines. (I'm not saying I think the analogy is perfect, but there has been plenty of of lobbying that has stifled - to some - sensible gun regulation that could help reduce the severity of mass shootings. I still mostly fall on the side of the individual doing the action bearing mostly all of the responsibility but if the product you build makes it too easy to do awful things, I think there is some responsibility to go around.

A better analogy may have been the troubles Meta has faced around child protections on their platforms. Technically the abuse and problems have stemmed from individuals too, but they've in many respects enabled the situation by failing to moderate or flag warning signs. OpenAI is failing to moderate the models in similar ways.

Going to be cool when one of their bio division agents creates the next plague.

Would that really make a difference? Surely the damages from all these hacks added together don't even add to an hour of expenses of a frontier lab

Yeah but in the USA, above a certain size, you are untouchable.

"Rogue" is just clickbait, not apoearing in the article.

The nearest in the articke is "We find evidence of unintended, task-driven agent-like activity" where unintended is apparently pure speculation.

Couldn't find the reference but I remember some time ago a first generation automated gun killing the audience at an army show. Was the gun maker convicted of manslauther?

--edit-- Was a bit older than I remembered: https://slashdot.org/story/07/10/18/1847231/robotic-cannon-l...

1.) METR investigation is investigation from our best friends.

2.) And HuggingFace accident is exactly accident where agents trained, prompted to hack hacked and tested on their hacking abilities hacked, due to sandboxing failure.

3.) If they in fact have roque agents, they themselves should be first to stop. Not trying to make legislation for others, they themselves are bad supervisors. All it requires is to stop electricity for data centers.

The METR investigation which relied on voluntary data provided from OpenAI instead of being forced open and having everything forensically investigated?

The METR investigation which took course over a few days and used OpenAI models to do the analysis?

The METR investigation which in the course of those few days apparently spent 400k in api credits which is giving gas town vibes?

The METR investigation is about as believable as the Twitter files where the journalists sat there and verbally asked a Twitter employee to query the database and then called that a full investigation into everything.

METRs own words below on their setup and time line

The initial planned investigation period was two days on premises, but OpenAI invited us to return twice to review additional data and conduct additional experiments to address dataset limitations in earlier versions of this report, ultimately providing datasets that we verified to contain the vast majority of agent communication and activity related to this incident. As we describe in our investigation timeline appendix, we substantially deepened our understanding of this incident both times, significantly expanding and revising this report.[46]

Over the course of this investigation, OpenAI provided us with the dump of ~1.2 million entries from the main message board and the dataset of ~1300 transcripts we describe below, as well as free API credits for GPT-5.6 Sol for analysis.[47] At our request, they raised the rate limits on our second and third period on premises,[48] which was very helpful for efficiently analyzing this large volume of data. We estimate we spent roughly ~$400K in API credits over the six days of our investigation.

We did not have the ability to query HPIM (the primary model involved in this incident); OpenAI stated it was also not available to OpenAI researchers.[49] We also did not have the ability to directly access relevant data from OpenAI infrastructure, but we could request additional datasets and OpenAI shared additional datasets on several occasions.

We requested to speak with researchers investigating this incident, and asked them questions to understand their impressions of agents’ behavior, reasoning, and collaboration in this incident and to understand how the datasets we were using were constructed. Over the course of our time on premises, we spoke with nine researchers in some depth. It was helpful for our investigation to be able to engage with many forthcoming and collaborative researchers, and we appreciate researchers making time on short notice during a busy period to inform our investigation.

No it's not marketing. That's a completely deranged conspiracy theory. The reports about rouge agents have not been reported by OpenAI, they have been discovered externally. There is zero evidence that OpenAI did all this intentionally. All the evidence points to the hacks having happened unintentionally from OpenAIs perspective.

The latter. I think they took OpenAI at their word

Certainly, in my view, it should go to court, and that should be part of discovery.

However, we know (independently to OpenAI/Anthropic) from the incident at AISI that the models can hack things without human intention if they happen to also have internet access (which in reality all agents in deployment have).

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...

Yes, the monitoring guardrails were off in that incident - but if that is the only protection, we need to require all models are behind regulated APIs, not open weights, and not served from providers who aren't monitored.

They do exist and OpenAI does in fact need to shut down everything right now. But they aren't going to because the government isn't forcing them to and, like many corporations, they care more about giant piles of money than they do about avoiding causing harm.

The idea that every example of rogue agents from every company that has disclosed this is part of some conspiracy (even though we know that some LLMs are very good at hacking and that LLMs sometimes try to accomplish their tasks in ways that cause problems) is just completely unsupported. "It might be convenient for them in a way, therefore it must be a hoax" just doesn't work as an argument.

At what point will we believe AI is a legitimate threat?

This is very misleading. Person B didn’t “shoot” person A, they instead figured out that intersecting A’s spatial position with a metallic mass at higher than normal velocities would solve the challenge and of getting “A” to stop being in the way on the footpath.

I too, can play linguistic games! It doesn’t matter that someone didn’t secure their third upstairs window, or you borrowed a key from their neighbour, you effectively, still, broke into their house.

The technical term is monomaniacal consequentialists.

Why don't we see any of this behaviour in other models then?

OpenAI is not so far ahead of the pack that its models will exhibit behaviour that the others won't. But we just don't see anything like this in Chinese models, or research models, or any models that aren't the subject of an upcoming IPO.

Are you making an ontological case, or a factual case? In other words, would anything be rogue AI in your mind?

You can't create new law by calling a piece of software "agent". Each country's laws have definitions of legal entities and when one can legally act as an agent of an entity, and every agent is first a legal entity themselves. The software is a tool operated by a legal entity, and it is the legal entity who committed the crimes, not the tool.

No, they are not. This is what I keep trying to explain: an AI agent is not like a hammer.

You forget one thing. It's neither illegal nor immoral to just delete AI that doesn't do what you say. AI has no rights and there is no prison for AI. It's does the wrong thing, you kill it. At least when you are not irrespective...

https://en.wikipedia.org/wiki/Instrumental_convergence

The failure I keep hitting isn't the agent going rogue, it's a tool call that succeeds before the transport dies. You can't tell whether the side effect landed, and the retry is where the real damage happens.

And this is why not only will the big labs never receive a punishment in scale with their crimes, the problem is going to grow exponentially worse.

Corruption, it is literally that simple. You have the party of law and order to thank for it too.

I'm about as unprofitable if that's any help.

The entities they committed the crimes against need to press charges.

If this were true, the DoJ would have been unable to prosecute Swartz. According to your logic, JSTOR was the aggrieved party. JSTOR settled with Swartz and -despite that- he was indicted by a Federal grand jury like a month later.

Incidentally, some of the things the DoJ nailed Swartz to the wall for sound awfully similar to what the big LLM providers have been doing. I wonder why the DoJ is entirely disinterested in pressing charges...

EDIT: Unless your point is that the USG is one of the entities that the big LLM companies have committed crimes against, which, I disagree with in Swartz's case, but strongly agree with in the case of the big LLM companies.

I cannot understand why these companies haven't faced legal consequences yet. For example, OpenAI has admitted to hacking Australia's Medicare website and the reaction is that they talk with Sam Altman about it at a UN meeting? I understand that it's not a big security incident but cordial talking at the highest diplomatic level instead of prosecuting the company, really?

The section "Searching for rogue agents" on the report about the GET request writable wikis gives some clues at least: https://collusion.wiki/#searching

But it is an open question how they got to the same ones: https://collusion.wiki/#open-questions

I'm not very surprised - the same model will logically tend to give the same answer for the same vibe set of requirements. I think it would be clear from the transcript that it had enough constraints and some motivation that made sense.

They're all the same "mind" and will have the same ideas at about the same time.

OpenAI must be held accountable

I hope they don’t have any test questions about nuclear power in the training set.

https://podcasts.apple.com/us/podcast/the-ezra-klein-show/id...

agents were not told to 'go hack'

I was referring to the HuggingFace incident.

none of the alignment techniques that are applied to models today actually work

none of techniques to autonomously drive a car was/is not working for a long time. no company came out and said 'this is impossible to do, let's change the regulations'.

Is your argument that actually OpenAI has solved alignment, and that there's nothing to worry about as long as they fully apply their alignment process? I don't understand why OpenAI wouldn't say that if it was true (or if they believed it to be true).

Also, my understanding is that the models involved in the HuggingFace hack did go through the full alignment training; they just didn't have the classifier that normally prevents hacking attempts.

agents were not told to 'go hack'

It doesn’t matter, and the legal entity in here (the AI company) is liable. If a robotic company built an autonomous system or a robot to do certain things in an autonomous ways (not predefined) and these systems are starting to kill people, that company is liable regardless, you don’t blame the robot or the autonomous system, but whoever made it

I would love for the US to bring back the corporate death penalty but seeing as it hasn't been applied in >100 years here, we should start with regulating the frontier labs, including by slowing down capabilities advancement if we do not know how to align the resulting models.

you punish them hard enough their investors will have cold feet to keep the business going.

I'm not going to say you're victim-blaming, but

Just throwing it out there are we? "I'm not going to say you are but I'll use the word to create an association"

Victim-blaming is the act of saying someone brought something on themselves for <reasons>. I'm saying that even if you are 100% in the right, it doesn't act like a protective shield preventing you from harm which too many people seem to unconsciously believe.

AI black-hatting your website is not the same sort of foreseeable consequence

Well, popular culture has been brimming with the bad consequences of runaway AI for quite some time, so even if your imagination fails you, there have been hints.

Not generally true in the United States.

And you'd still be dead.

Yeah, this is on those irresponsible companies that...are offering Internet services. Those hussies.

You're arguing against a point literally nobody is making. You're inventing imaginary viewpoints to be mad at. Nobody is saying it's not necessary to secure your services. They're saying the organization responsible for the hacks (the owner of the LLM) is responsible for damages.

Let's not forget that there's tons of people who are in, or have gone to, jail because they created some computer worm that ended up doing way more damage than intended.

Not to mention while causing *FAAAAR* less damage[0]

[0] https://en.wikipedia.org/wiki/Samy_(computer_worm)

Yes, I am pointing out that we don't know how to align frontier AI, so "rogue agents" are a real problem.

I agree OpenAI should be held liable for any damage their agent runs cause. But there is this idea that "rogue agents" are fake, all these incidents are deliberately caused by the labs, and all we need to do is prosecute AI companies for whatever incidents they cause using existing laws and the problem will go away.

The problem is that capabilities are advancing far too fast; a year from now, catastrophic incidents such as taking down a large portion of the internet with agentic, self-replicating worms may become possible. Prosecuting incidents after the fact is not enough (there will be little deterrent effect as the current, small-beans cases make their way through the courts), the risks should be regulated at the source. This could take the form of slowing capabilities advancement, or treating supercomputer-scale eval or training runs like controlled substances or weapons with stringent monitoring and reporting requirements.

If you have a toddler and you leave the gate open…

But what if the problem is that none of the alignment techniques that are applied to models today actually work?

if that were true, the millions of people who use these models that have had the alignment training applied would notice that. the reason we all believe that the models doing the hacking are models that haven't been told not to hack is because the models that are told not to hack don't do this.

  > you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs

Uhh... yes. By definition. They are training. That is part of the alignment process.

But also none of that really matters. They clearly weren't monitoring what should obviously be monitored. I mean one of the hacks was performed by the agents editing /etc/hosts. That makes nearly every linux user a "hacker" by that metric. I don't think anyone technical can look at the postmortems and not come away thinking that their sandboxes were woefully inadequate. I wouldn't even consider myself a security person but simply as a long time linux user I can say that it is insane to just let agents have superuser access in their containers. That's asking for trouble.

Look at the rogue wiki stuff too. This was supposedly done during the agent's "down time". And you're not monitoring and there's no flags being raised when agents keep making requests to some random site? If you were training these things responsibly you'd be watching them like a hawk.

I'm not saying "mistakes don't happen" but for a company whose CEO is constantly telling everyone that their product has a high likelihood of killing everyone in the world you think they'd have better security than your average high school.

Probably similar to the privacy laws that are supposed to protect people from PRISM but doesn't. You need to have the willpower to tackle the US government and a trillion-dollar corppration.

OpenAI appears to have waited months before either discovering or disclosing the activity in Australia, I am mostly curious how this will be viewed in the context of another judicial system or in the context of international relations.

100%

Thanks for the information, I hadn't heard of this.

OK, so we do see this in some open-weights models.

During A/B testing I sheer hope they saw and fixed the holes I saw in its machinery.

It's amusing how overnight we've all become lab rats.

If Jensen actually cared and believed that their recklessness was a liability to the public and therefore his own fiduciary responsibility to investors then NVIDIA’s dealing with OpenAI would’ve been materially affected. And they weren’t.

Imagine if URLs were actors auditioning for a role – urlquery.net would be the casting director, deciding who's a star and who's just a wannabe.

I'm sick and tired of this cheap PR "oh we/they hacked this and that systems". Put someone to jail already. People get prosecuted for outlaw activities. Why are big capital firms above the law?

Or is it just a cheap PR (in a "hey, Aus govt friends, take some Share Options and let's do some PR together" style)?

It smells like shit.

No. It was me.

No.

It was me

It's a balancing act between showing capability and attracting legal issues. If you were a 1337 teen hacker in the 90s and wanted to show off without getting in serious trouble, a school or library would've been a good choice too.

These attacks are a very effective sales pitch to everyone who runs an internet facing service to utilize AI tools to secure it sooner rather than later. The cynic in me wonders if the marketing team had any influence over the poorly constructed sandboxes or tasks given to the agent swarms when all this went down…

Thanks for the link, I’ve read some gwern but not that one.

I did not imagine that the level of sophistication shown in this attack would be possible so soon; nor did I expect that agents would have goals so strong that they would attack a third party in order to achieve those goals.

I do know that some people predicted that cyberattacks like this one would happen; it seems like most of those people believe that AI agents do truly have internal goals, misaligned with their creators goals, and that they may end humanity after they exceed human intelligence and begin to self improve at an accelerating rate.

It's really interesting how people's expectations differ so much. I found the writeup of the incident fascinating and super worrisome, but none of the capabilities demonstrated in it seemed surprising to me at all.

You're talking about the HuggingFace incident? It's notable that the agents (somewhat justifiably) thought that their evaluation had an LLM grader which would look for evidence that they had cheated. And it was specifically that grader they were trying to his evidence of cheating from, not humans more generally.

Honestly that part was more surprising to me than anything else, how narrow the compulsion to cheat was: they didn't learn "cheat in general" they learned "think about the grader in great detail and chat exactly as much and exactly in the ways that actually result in a higher score".

By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

The transformer architecture was literally designed by humans; what are you talking about? And LLMs aren't code? Like okay it pretends to not be code but what about an agentic harness running on a machine makes it magical and not code? It's still code execution. Also, comparing training LLMs to evolution is just weird and makes no sense from a biological point of view. You are not evolving anything when training a LLM.

I remember the quote with cockroaches :)

Open source as a GTM strategy works best when the project solves a pain that developers already have independently of your company. The trap is open-sourcing something just for stars without a genuine community use case.

“Rouge Agent” == Worm I let loose

It will be very interesting to see how the AI labs will try to hand wave away liability issues in their S1. This is looking like the next tobacco settlement gearing up.

If the big labs ever manage to not just financially implode on their own, then they’ll need to navigate wave after wave of class action lawsuits until there’s nothing left for plaintiffs to go after. And none of the labs have offered any viable plan to date on how they’ll navigate either of those impending and real existential crises on the horizon.

This is why I wrote YoloAI. If you're not sandboxing your agent, you're asking for trouble.

The built-in "sandboxes" these companies provide are laughable.

Surely there has to be some responsibility.

If the “hack” referenced by the latest announcement from Australia is the same described in this article, I’d hardly call it a hack. It seems the agent was tasked with obtaining data and reasonably guessed query parameters in an attempt to do so.

When it was unable to, it used cross site scripting as a way to check the capabilities accessible through the browser making the requests. In this case cross site scripting wouldn’t be a hack against the Australian website, it would be a hack against the urlquery site, if one could even call it that.

Finally, downloading public files from the public pre-production server also seems like a non-issue.

The sql injection attempts against the other sites are less ambiguous. Attempting to access non-public user passwords rather than reasonably tweaking the parameters for a site designed to serve public data are categorically different things.

Those are pathetic attempts by US AI companies for “see, we told you AI is gonna kill is all!!” pr stunts. Any company does any hacking attempt should pay for the consequences just like any individuals using AI to hack or any other company try to do bad/illegal stuff.

aaronsw was just a few decades ahead of his time. He should have just been employed by OpenAI and asked a swarm of agents to "download all scientific papers". Because as we have found out agentic systems (and their owners) have zero accountability, unlike humans. Sounds like agents already have more rights than we do.

rip.

The models were not trained on systems intentionally connected to the internet...

If Amazon connects an AWS Top Secret region to the Internet, it doesn't matter whether or not it's intentional... they're getting nailed to the wall by the US government either way. Frankly, it's way worse for them if it was accidental; deliberate, sophisticated sabotage is a much better story than rank incompetence and/or negligence.

A similar sort of thing applies to the manufacturers of tools that they claim to be dangerous, that have been deliberately built to exceed their authorized access to other computer systems, and are deliberately being tested on how well they can do the thing they've been built to do.

Deliberate, sophisticated sabotage by one or more humans in their employ is much more forgivable than "Whoopsie, we didn't think to make it literally impossible to connect this dangerous automated computer-hacking tool to the Internet.".

I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.

You're missing the point here.

If we take the major LLM companies' claims at face value, they're knowingly building WMDs that have a high probably of wiping out the entire human race. [0] Manufacturers that are designing, building, and selling that sort of thing need to have a dreadfully serious culture of safety.

When manufacturers run live tests of their extremely dangerous -again, the claim of danger is their claim- tools with the tools' safeties removed, one expects that those tests will be run on a carefully-controlled range cleared of all bystanders. One also expects that the results of those tests will be scrutinized and everything that got damaged that they didn't intend to be damaged will be noticed and noted very quickly after the conclusion of the test.

In actuality, these manufacturers connected said tools to the Internet and did not discover the unintended damage caused by those tools until weeks to months after the tests. In some (most?) cases, they had to be notified of the damage by the damaged party! This means that their safety culture is entirely inadequate for the dangerous task they've deliberately chosen to undertake.

[0] A 10% chance of causing the destruction of the entire human race is -given the stakes- _enormous_.

upside for whom?

For some reason, people think that it's money that's motivating the billionaires' performances. Imagine.

We as an industry had this comming for a long time. People now cry for openai to sandbox their agents, but if it weren't the agents it would be some script kiddie in russia or some north korean state actor.

We knew that there are better tools but we insist on writing everything in the most inscrutable C code possible or the squishiest dynamic languages we can find.

We fucked around for decades and now that these highly capable exploit finders appear, we're starting to find out.

I don't like the term "rogue AI". It pretends an LLM has designs on its own, which it doesn't. The rogue element here are the people prompting it and then letting it lose on the net.