Agreed we should ban the term rogue for agents. This implies a moral compass that is not there. They were directed to find data without guardrails or limits, it is not rogue it is intended.
AFAIK all of these incidents happened when OpenAI contracted out to a company called Irregular (https://www.irregular.com/) to run these sandboxed CyberGym tests. They all happened around Mar-June and seem to be from the same collection of agent trials. Since then they already released Astra. Halting now is likely just a way to manage blowback.
This should be the top comment on every one of these godforsaken posts. I don't want to see a single report about OpenAI hacking the UN until Sam Altman addresses the role Irregular played in these attacks. If he can't provide an honest postmortum concerning their business partners, then he's proving why nobody trusts him.
It seems like anthropic is far ahead of openai, and has no reports like this. We have to conclude this is a skill issue/engineering quality problem inside openai.
just because they are a well known name, doesnt mean they havent botched hiring over the last two years or so
Ive anecdotally heard that openai is far more chaotic, which includes not having a central infra team for example (or at least some teams not counting on depending on them). At least the previous hacks in openai were mainly due to bad infra architecture design.
Are Chinese models actually also doing unexpected things, hacking (intentionally or unintentionally), etc but there is just zero transparency provided when it happens?
To use an analogy to another industry, if you had US food companies providing reports whenever their food had issues, even if it was just during testing or training phases… and then also had a bunch of Chinese companies but who never reported having any food issues…
there is just zero transparency provided when it happens?
Neither does openai, as there keep coming third-party reports of incidents that have happened there that openai either did not know or basically concealed.
Generally, the Chinese don't have a good track record of supressing information.
As in they do the 'we have deleted tons of videos and posts about the thing that didn't happen last week', but it seems they haven't really managed to transcribe 'Streisand' into Han characters so far.
because they didn't burn obscene amounts of Money training each LLM and didn't promise half the planet that their business is worth a trillion while not having even operational break even let alone the cost if you include the overhead.
Soft Bank just raised couple of billions in junk bond sale to support open-ai's current operations before the IPO.
it's a crazy situation where on one side the Chinese / open source LLMs are catching up and reducing the token price, on the other hand the current leading labs have spent everything they got, every new model will cost much more and the public market is too shaky to support an IPO.
They will make it, I don't doubt it, but it's a crazy situation.
Chinese models might do the same. But they just don't lock thousand monkeys in basement and come check result week or more later...
It is entirely possible that they run stuff in more responsible matter. Especially as there is stronger culture of oversight and personal responsibility than in west where such culture does not exist.
Because Chinese investments are not so encumbered by changes in the US treasury interest rate. Also, China doesn't spend so much money for chasing model performance.
I firmly believe that this is just the public-facing story here.
Stopping AI development and research, even slowing it, would be a disaster for the SOTA companies and their first-mover advantage.
There’s almost no way to coordinate this across the world. Zero chance that everyone stops. We can’t even agree to coordinate on weapons tech that’s decades old with zero “everyday joe” impact.
I don't believe a word coming from them. As I see it, this is happening because the money for training models has dried out. The treasury interest rate risings tells you all you need to know.
I think any argument that this is a cynical attempt at regulatory capture is destroyed by this; the economic incentives of releasing more capable models are too large. I might be persuaded that they are actually running out of money, and this is really just a cover for reducing burn..
I welcome this though, I think the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjust. More intelligence isn't necessary for meaningful impact and the risks that are obvious and present and unsolved aren't worth the cost benefit analysis.
related https://news.ycombinator.com/item?id=49868083
Agreed we should ban the term rogue for agents. This implies a moral compass that is not there. They were directed to find data without guardrails or limits, it is not rogue it is intended.
"they were given the hacking test, what did OAI expect?"
"they didn't watch it, they didn't stop it when they first became aware"
"are we going to defer to the same valley elite that brought us algos and social media?"
statements normies are using and resonating with
AFAIK all of these incidents happened when OpenAI contracted out to a company called Irregular (https://www.irregular.com/) to run these sandboxed CyberGym tests. They all happened around Mar-June and seem to be from the same collection of agent trials. Since then they already released Astra. Halting now is likely just a way to manage blowback.
This should be the top comment on every one of these godforsaken posts. I don't want to see a single report about OpenAI hacking the UN until Sam Altman addresses the role Irregular played in these attacks. If he can't provide an honest postmortum concerning their business partners, then he's proving why nobody trusts him.
Ah, so there was a solution to hinder the big-bad AI after all..? Simply... Turn them off?
It seems like anthropic is far ahead of openai, and has no reports like this. We have to conclude this is a skill issue/engineering quality problem inside openai.
just because they are a well known name, doesnt mean they havent botched hiring over the last two years or so
Less bad, but https://www.anthropic.com/news/investigating-incidents-cyber...
In some sense though, sure, skill issue explains the gap vs. Anthropic’s much less severe alignment issues.
im not sure why more people arent calling it out.
I, for one, have shifted from being policeman to creator and explorer. It is so much more rewarding and less stressful.
Anthropic's models seem crippled and hamstrung to begin with
have you tried opus 5.5? anthropic is way ahead, atleast in terms of publicly available models
Have you tried Astra? Way ahead?
opus 5.5 > fable 5.1 >> astra
astra is a good workhorse, but its much less generally intelligent
Opus 5.5 does seem competitive with/better than Astra and is more affordable so usage doesn’t run out so fast
Slot machine users argue about which machine pays better
Hint. You lose using either.
What do you mean? https://www.felonybench.com/
They're almost tied for felonies.
That’s not the most parsimonious explanation even if the assumption it rests on (anthropic ahead of OpenAI) is true, which we don’t have proof of.
Ive anecdotally heard that openai is far more chaotic, which includes not having a central infra team for example (or at least some teams not counting on depending on them). At least the previous hacks in openai were mainly due to bad infra architecture design.
Why does China not have this problem?
Are Chinese models actually also doing unexpected things, hacking (intentionally or unintentionally), etc but there is just zero transparency provided when it happens?
To use an analogy to another industry, if you had US food companies providing reports whenever their food had issues, even if it was just during testing or training phases… and then also had a bunch of Chinese companies but who never reported having any food issues…
the target companies are the ones reporting the hacks in many cases.
Neither does openai, as there keep coming third-party reports of incidents that have happened there that openai either did not know or basically concealed.
Generally, the Chinese don't have a good track record of supressing information.
As in they do the 'we have deleted tons of videos and posts about the thing that didn't happen last week', but it seems they haven't really managed to transcribe 'Streisand' into Han characters so far.
because they didn't burn obscene amounts of Money training each LLM and didn't promise half the planet that their business is worth a trillion while not having even operational break even let alone the cost if you include the overhead.
Soft Bank just raised couple of billions in junk bond sale to support open-ai's current operations before the IPO.
it's a crazy situation where on one side the Chinese / open source LLMs are catching up and reducing the token price, on the other hand the current leading labs have spent everything they got, every new model will cost much more and the public market is too shaky to support an IPO.
They will make it, I don't doubt it, but it's a crazy situation.
Chinese models might do the same. But they just don't lock thousand monkeys in basement and come check result week or more later...
It is entirely possible that they run stuff in more responsible matter. Especially as there is stronger culture of oversight and personal responsibility than in west where such culture does not exist.
Because Chinese investments are not so encumbered by changes in the US treasury interest rate. Also, China doesn't spend so much money for chasing model performance.
‘There are no “rogue” AI agents’
https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-age...
I firmly believe that this is just the public-facing story here.
Stopping AI development and research, even slowing it, would be a disaster for the SOTA companies and their first-mover advantage.
There’s almost no way to coordinate this across the world. Zero chance that everyone stops. We can’t even agree to coordinate on weapons tech that’s decades old with zero “everyday joe” impact.
I don't believe a word coming from them. As I see it, this is happening because the money for training models has dried out. The treasury interest rate risings tells you all you need to know.
The Huggingface hack occurred during reinforcement learning. Why can't they pull the Ethernet plugs?
The answer is probably: The newer models rely so much on stealing content in real time from the internet that training needs network access.
I think any argument that this is a cynical attempt at regulatory capture is destroyed by this; the economic incentives of releasing more capable models are too large. I might be persuaded that they are actually running out of money, and this is really just a cover for reducing burn..
I welcome this though, I think the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjust. More intelligence isn't necessary for meaningful impact and the risks that are obvious and present and unsolved aren't worth the cost benefit analysis.