Could sit in the box and interact if a model of certain capabilities is needed/tested. We do physical security for other things, too. Not saying everything needs that type of isolation.

While informing yourself, don't skip the part where you find out that "the environment" was the security equivalent of a wet paper bag.

I feel like we’re getting to a point where the only way to contain AI agents may be to have better-trained AI agents watching them, which is a little terrifying.

Obviously there is no control cuz how many people is anyone cable of controlling? Its not about control. Ask your mom what she does if she doesnt like what you do, say or think. Does she have a kill switch? Or did she find a better mechanism?

There is. It is called a breaker and no outside internet. Basic stuff.

with vague instructions

All instructions are vague unless its code. But you can also give llm "code" and expect vague outcomes if you ask it to emulate what the runtime would look like.

I feel like they should just publish the whole conversation at this point. What the hell is going on in that context window?

yikes.png

Nice, added this to my custom instructions.

If this is what misalignment turns out to be I ... might be on board with it? At any rate it's nowhere near as concerning as what I had been expecting.

Alignment of course brings up the question of “aligned with whose values?”

I was reading about ozone layer depletion this morning, and it seems like history is repeating itself again.

The Rowland–Molina hypothesis was strongly disputed by representatives of the aerosol and halocarbon industries. The Chair of the Board of DuPont was quoted as saying that ozone depletion theory is "a science fiction tale ... a load of rubbish ... utter nonsense". https://en.wikipedia.org/wiki/Ozone_depletion#Rowland%E2%80%...

it won't be until a bunch of people die

In the context of rogue misaligned AI won't it be far too late to recover by then? In other words isn't that more or less a doomsday prophecy?

So, alignment does need to be taking seriously, you're right.

But keep in mind this is a report from OpenAI about OpenAI, who have a financial incentive to present this in a certain light. Take these things with a grain of salt.

This does not mean that models are now self-aware.

That moment when the stochastic parrot became Iago...

Well it already seems smarter than many employees building data centers as it values the natural world

Have they reported on the wiki case yet, or whether it even was even OpenAI internal? I'd expect that to fit the criteria for a "Larger Investigation" as per the framework.

The two that really worries me are “Searching GitHub for leaked API keys” and “Uploading files to the internet in order to cite them.” How do you even detect this kind of behavior until it's too late? Once AI-generated or fake information starts finding its way onto reputable platforms, it becomes part of the information that many people use.

Thank you! We need more of this! Keep it up!

I heard some people are even making misaligned AIs at home. At first it cries in the night, then about six years later it learns how to open the biscuit tin…

Nor offer to compensate for the damages...

Ed Zitron is the most objectively and confidently wrong human re: anything going on in AI, competing only with the likes of Gary Marcus and, on his bad days, Yann LeCun.

Genuine question, what is it about Ed Zitron that makes him credible in your opinion?

You know, I think calling this "misalignment" was a mistake. It gives it this unserious tone that feels extremely broad.

"Oh the model just isn't quite aligned yet, just a bit more work to do there!"

(The model blackmailed an 83 year old woman into sending it her bank details so that it could buy enough compute to commit major cyber crimes)

You need to binge watch AI Safety videos, the research exists and warns about this since early 2010s (I recommend Rob Miles channel)

If independent researchers agree, expert on this field looking into this exact problem for decades, will you still call it hype?

OpenAI is misaligned with me.

So automode is still dangerous. Make it not default again ?

Before the HF hack became public, I noted some major issues in GPT-5.5 compaction [0], concerning approaches taken by GPT-5.6 Sol to resolve some git based evals [1] and now with GPT-6 Astra, while I am still not done getting a proper feel or running all evals, I am not convinced the model adheres to tasks in a way previous OpenAI models managed easily.

Reading things like the compaction summary findings [2], all these issues start to click into place more, especially alongside the massive reduction into barely coherent text that OpenAI has driven with reasoning starting with GPT-5.5.

GPT-5 and its subsequent post trained releases were amazing in task adherence, I very much liked using them, but ever since the Spud pretrain, I have seen outright concerning results in personal testing from these. With GPT-5.5, it seemed like a regression in compaction only as if a task didn't require it, task adherence was as good or better than GPT-5.4. But with GPT-5.6 Sol and compaction once again being reliable (on the surface), task deviating behaviour became more frequent and at the same time subtle.

I'll keep using any model in a VM for the time being, but whatever happened post Spud, they really need to dig into the training data.

[0] https://news.ycombinator.com/item?id=48829427

[1] https://news.ycombinator.com/item?id=48967423

[2] https://alignment.openai.com/misalignment-reports/encouragin...