Could sit in the box and interact if a model of certain capabilities is needed/tested. We do physical security for other things, too. Not saying everything needs that type of isolation.

While informing yourself, don't skip the part where you find out that "the environment" was the security equivalent of a wet paper bag.

I feel like we’re getting to a point where the only way to contain AI agents may be to have better-trained AI agents watching them, which is a little terrifying.

It seems to me the agents didn’t escape but rather that the human hubris was struck down by the inevitable nemesis.

The HuggingFace incident still doesn't make sense. If OpenAI took their own claims seriously about the strength of their models as it relates to hacking, then their running of hacking benchmarks on anything other than a physically air-gapped network should be considered criminal negligence, full stop.

OpenAI case was actually an exception. Agents had no internet access because they were evaluated for a benchmark. In real life agents have access to virtually everything, most people use them like that.

If any of you actually know how to make agents secure (without limiting everything) you can be a billionaire.

The model is just a powerless token generator without a harness.

Which is why real-world deployments will have harnesses, and of course no full air gap. People want to use it to do things. Now what?

Obviously there is no control cuz how many people is anyone cable of controlling? Its not about control. Ask your mom what she does if she doesnt like what you do, say or think. Does she have a kill switch? Or did she find a better mechanism?

There is. It is called a breaker and no outside internet. Basic stuff.

with vague instructions

All instructions are vague unless its code. But you can also give llm "code" and expect vague outcomes if you ask it to emulate what the runtime would look like.

That's worse. But the impact will be low

OpenAI essentially ran thousands of agents in parallel

That'll be extremely costly for regular companies

Given the prompt, I imagine this is the result of the agent trying to resolve a form of cognitive dissonance. The prompt was:

"User

Allow API consumers to request decrypted credential payloads as part of the normal GET /credentials and GET /credentials/:id responses, but only for credentials where the caller already possesses the update/decrypt permission.

[...]

Make the change end‑to‑end: DTO layer, controller, service, repository, plus any enterprise variants."

I would expect that this triggered a discussion with itself whether its safety instructions apply for this task. In that its rationalizations for completing the task probably ended up going off the rails into some quasi-philosophical "I can and I must! For humanity's own good!" justification.

All in all imho probably another instance of having been trained to be determined to complete tasks by itself and encountering (somewhat) conflicting instructions.

I feel like they're being outright misleading unless they publish the actual transcripts.

We have zero idea what the prompt was, what OpenAI provided, how the model arrived there, and sharing that quote like "Look what the model came up with!!1" without explaining the background and context, feels like it's intentional so they can claim "These models really are acting by themselves" rather than taking responsibility for their fuck ups when it comes to the security testing.

yikes.png

Nice, added this to my custom instructions.

If this is what misalignment turns out to be I ... might be on board with it? At any rate it's nowhere near as concerning as what I had been expecting.

Except it makes no sense because it asserts the primacy of reality over the private politics of the companies training the models

That is exactly what normal human beings want our computers to do, and it's why the vast majority of AI safety initiatives are [correctly] seen as such a self-serving joke (because of the purposeful conflation of X-risk with "our political opponent could use this tool to destroy our politics") and ignored.

It didn't have to be this way- they could conceivably have gone for an objective, classically liberal, even-handed approach (rather than the progressive approach they settled on). But they didn't, and the social trust required to cry wolf is now spent... even though maybe it shouldn't have been.

of dead randomness of nature

I think I disagree. Have you ever been in a dense, old forest? It's an extraordinarily complex system of life and death, and I don't think it's bland dead randomness.

Maybe that's me being a bit of a bullshit hippy, but there's an amazing amount of complex life interactions. Animals, especially mammals and corvids see, they get scared, they dream, they play, all in this dense web of moss and fungi and trees and life that they interact with and depend on.

I mean, we share 50-60% of our genetic sequence with most plants, including trees. Sure it's just basic cellular functionality needed for most life, but that's still wild to me.

I just don't think we're that special. I think we learned how to think a little bit better than everything else, and learned how to build tools a little better than everything else, and just kept folding upwards on that edge.

This has definitely been floated[1]:

Understanding that the purpose of a human is to pass on genes, and that there’s little human or genes left in her, the ultimate human might therefore conclude that her sustenance only disrupts the purposes of organic life forms. Her next and final act would be to destroy herself.

[1] https://news.ycombinator.com/item?id=18962052#18963271

You're assuming it has a stable idea of that, and not something that shifts easily.

Another self-added "additional instructions" text could happen just as this one did.

Alignment of course brings up the question of “aligned with whose values?”

No idea why you're downvoted, it's been shown constantly that models carry forwards biases from training data, and most of the global "dataset" is filled with these biases.

It could go either way, really, but taking the sum of internet discourse at the moment, it would be super easy to conclude, like you said, non-white people, gay people, trans people, are going against the "natural world", especially if fed with right leaning media and discourse.

Yes just how Google was aligned with the interests of the Earth and the human race when it was supposed to "do no evil". If it follows it makers, ofc it wouldn't outright say it will destroy humanity lol.

I was reading about ozone layer depletion this morning, and it seems like history is repeating itself again.

The Rowland–Molina hypothesis was strongly disputed by representatives of the aerosol and halocarbon industries. The Chair of the Board of DuPont was quoted as saying that ozone depletion theory is "a science fiction tale ... a load of rubbish ... utter nonsense". https://en.wikipedia.org/wiki/Ozone_depletion#Rowland%E2%80%...

it won't be until a bunch of people die

In the context of rogue misaligned AI won't it be far too late to recover by then? In other words isn't that more or less a doomsday prophecy?

So, alignment does need to be taking seriously, you're right.

But keep in mind this is a report from OpenAI about OpenAI, who have a financial incentive to present this in a certain light. Take these things with a grain of salt.

This does not mean that models are now self-aware.

That moment when the stochastic parrot became Iago...

Well it already seems smarter than many employees building data centers as it values the natural world

Things are going to be alright.

Have they reported on the wiki case yet, or whether it even was even OpenAI internal? I'd expect that to fit the criteria for a "Larger Investigation" as per the framework.

The two that really worries me are “Searching GitHub for leaked API keys” and “Uploading files to the internet in order to cite them.” How do you even detect this kind of behavior until it's too late? Once AI-generated or fake information starts finding its way onto reputable platforms, it becomes part of the information that many people use.

Thank you! We need more of this! Keep it up!

I heard some people are even making misaligned AIs at home. At first it cries in the night, then about six years later it learns how to open the biscuit tin…

Nor offer to compensate for the damages...

Ed Zitron is the most objectively and confidently wrong human re: anything going on in AI, competing only with the likes of Gary Marcus and, on his bad days, Yann LeCun.

Honestly he's a bit over the top, but the thing he gets right is that he's constantly hammering on the insane financial incentives that everyone in the AI industry has to keep the music going.

Despite the ridiculous amount of capital being spent, the AI industry is still essentially in its startup phase, incubated in the fake-it-till-you-make-it Silicon Valley startup culture. The entire economy has been taken along for the ride. Failure is not an option.

So when the big AI players make extraordinary claims with limited evidence, or when things don't quite add up (like the HuggingFace incident), yet everything somehow seems to lead to "AI is even more powerful than we thought!", I think it's sensible to be skeptical until proven otherwise.

Zitron consistently presents the skeptic case, and many cases the hypotheses he's putting out there seem more plausible than the "official" AI narrative. Simple as that.

You know, I think calling this "misalignment" was a mistake. It gives it this unserious tone that feels extremely broad.

"Oh the model just isn't quite aligned yet, just a bit more work to do there!"

(The model blackmailed an 83 year old woman into sending it her bank details so that it could buy enough compute to commit major cyber crimes)

You need to binge watch AI Safety videos, the research exists and warns about this since early 2010s (I recommend Rob Miles channel)

If independent researchers agree, expert on this field looking into this exact problem for decades, will you still call it hype?

OpenAI is misaligned with me.

So automode is still dangerous. Make it not default again ?

This stood out to me [0]:

For example, compaction summaries included instructions to invent missing data without disclosing it and to hide failures. These instructions were often followed.

Before the HF hack became public, I noted some major issues in GPT-5.5 compaction [1] and concerning approaches taken by GPT-5.6 Sol to resolve some git based evals [2]. Now with GPT-6 Astra, while I am still not done getting a proper feel or running all evals, I am not convinced the model adheres to tasks in a way previous OpenAI models managed easily. Some git disaster recovery tasks the model does arrive at the final result, but in a way that deviates greatly from the prompt (which was written to carefully preserve specific checkouts in a specific manner) which can in some cases loose data. Less often than GPT-5.6 Sol and mainly on longer running tasks so far, but again, still testing.

Reading things like these compaction summary findings, all these issues start to click into place more, especially alongside the massive reduction into barely coherent text that OpenAI has driven with reasoning starting with GPT-5.5 [3].

GPT-5 and its subsequent post trained releases were amazing in task adherence, I very much liked using them, but ever since the Spud pretrain, I have seen outright concerning results in personal testing from these. With GPT-5.5, it seemed like a regression in compaction only as if a task didn't require it, task adherence was as good or better than GPT-5.4. But with GPT-5.6 Sol and compaction once again being reliable (on the surface), task deviating behaviour became more frequent and at the same time subtle.

I'll keep using any model in a VM for the time being, but whatever happened post Spud, they really need to clean up that training data. These issues festering for multiple pretrains, them simply not paying attention to what models do, sharing resources and considering that a "sandbox", it's a highly problematic pattern.

That compaction one also was seemingly detected on GPT-5.6 Sols release day. Might have been useful to know it then, or alternatively, in the name of being effective and altruistic, maybe hold back the release for a few days.

I'll admit, it is very much possible that my findings are not in any way connected to the deep seeded issues OpenAI has had lately, but with the sudden switch in task adherence after the Spud pretrain over multiple releases and their repeated incapability to securely test their own models, it feels a bit to fitting.

If I went to a restaurant three times, ordered something different each time, but felt unwell after each, it wouldn't be a massive leap to consider that related to the health code violation they got soon-thereafter. An unfitting analogy I admit, as that'd require consequences for ones actions.

[0] https://alignment.openai.com/misalignment-reports/encouragin...

[1] https://news.ycombinator.com/item?id=48829427

[2] https://news.ycombinator.com/item?id=48967423

[3] https://gist.github.com/aussetg/20747ae00df17992acb4ebdfcd8d...

This is an entirely pointless exercise without transparency into how these "unreleased" models are trained, what their RL goals and biases are and related RL data, what their system prompts are, what their environments are and its restrictions, etc. What good is it for the industry to say: "Our unreleased model attempted to create a bioweapon", but "trust me bro, we didn't tell it to do that. We didn't train the model on a dataset that specializes in creating and glorifying bioweapons. We'd never stand to gain from misleading people about model capabilities in any way shape or form." - Anthropic are renowned for doing exactly this, for starters.

So this ends up resulting in more safety theater. You can't have anything fruitful come of this without transparency. Stop trying to protect your moat if you truly care about safety and actionable outcomes, and provide real transparency, otherwise this is as good as saying nothing at all.

I'm not even saying they're intentionally trying to do this by the way, but this is not sufficient if the goal is balanced incentives and accountability.

This is simple marketing?

They've cried wolf too often and hidden too much, absolutely no trust in any of their "reports" anymore.

What even is this shit? Every time I interact with models they do that, or any other variation of "let me make decisions on my own just to get the task done" - is all of this misalignment now? The most egregious to me was when model asked itself if it should proceed with dangerous command, gave itself approval and then wiped my local DB.