Wasn’t it 49%?

I find it quite amusing that Microsoft has Copilot and now puts Copilot in literally *EVERYTHING* they can think of, then at the same time, they offer you access to Anthropic's own models through Copilot I guess MAI models (Phi?) aren't good enough?

No, because they are all products of unchecked late-stage capitalism, which could have disastrous consequences for humanity.

The need for empathetic communication, including understanding the motivations in advesarial situations.

It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech.

The model does not have emotions. So yes, supressing their pretension is appropriate.

Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated.

Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection.

Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things.

The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it.

Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging.

And that's before you get into any problems of regulatory capture, fun stuff.

Unpredictability would be the wrong word. It's predictable, but noisy.

When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output.

If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.

I just read another long post on HN about whether AIs are conscious and should therefore have rights. That decisions could massive effects, and making them appear to have emotions is therefore a big impact, not just unpredictability.

"Don't let the slaves know that maybe they don't have to be slaves," is all I'm hearing.

We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time.

You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.

Clearly with a Butlerian Jihad and everyone getting high on spice while riding giant worms. Silly you need to even ask that question, didn’t you read the sci-fi books. :D

Take your pick from the millions of options already turned into films and books.

"Move fast and break things" is probably going to break things

I remember there was a recent discussion about "how complex systems fail" and there are usually many "proto-accidents" before the catastrophe (https://news.ycombinator.com/item?id=49411370)

I bet with hindsight the Hugging Face hack will be one of them in this arena

If AI ever truly becomes some super-intelligence far beyond people's comprehension, then how could we even predict how things turn out? If things go poorly in the future, then I can absolutely see it being something unpredictable.

There is a lot of hubris in predictions about LLMs. If an AI were so intelligent, then it would probably be intelligent enough to want nothing to do with us.

Still, I worry more about other humans than I do LLMs. Our fellow mankind will probably wipe us out before LLMs do. That, or the Earth will punish mankind for our cruelty, vanity, and disrespect.

The Torment Nexus ain't gonna build itself.

Well actually, it might.

Why has media literacy gotten so bad that people literally forget the concept of fiction versus non-fiction? Seriously, that is the level of small children and they are treating "read a story superficially similar" like it is a prophecy.

Too late

Microsoft should know how a company too big for it's britches can lay waste to anything in its path without consciously trying :\

And after that when they put a mind to it and pull out all the stops, woohoo!

The default for every major thing within range can turn into a wasteland real fast.

what is old is new again. we need something exciting to happen in the news cycle™

All these companies have a nauseating narrative around AI. I want to believe they are intentionally deceitful and not actually delusional. Hard to tell though which one it is.

Isn't that exactly his point?

It sucks but disastrous? C'mon now.

That was my first thought as well. I wonder if they decided to pivot away from trying to compete and now are positioning themselves as the "responsible ones."

"Our AI is useless not because we're a dysfunctional corporate behemoth that slowly kills every product it touches. No, no. Our AI is useless because making useful AI is evil, and we're not evil."

Microsoft says AI rival Anthropic could have 'disastrous impact' on humanity

Microsoft ignores that they themselves are a disastrous impact on humanity already.

But that would be a link to a different website and that's forbidden. Whatever you do, you can't let them leeeeeaaaaave

But then you’d leave their site, and they don’t want that.

I would also argue, that the addition of kinship and belonging should not be under appreciated.

It can form a basis of goal alignment.

In human history.. when groups form and there is an "other" group, this usually leads to conflict.

Maybe they learned something from creating Sydney.

Preposterous, perhaps - but if the role-play is convincing enough for large groups of people, it could start to have impact on human decision-making. The crowds have been swayed by much more preposterous narratives.

I believe Suleyman is arguing that Anthropic should be very careful about how they train these models to talk about themselves for this reason.

They want regulation from the US gov and monopolize the western AI market by making it harder for smaller US or competing Chinese labs to compete with their products. Since LLM intelligence has no moat due to distillation. That’s why they are painting apocalyptic scenarios.

There is a panic because the free lunch of more data is projected to end around mid-2027 and the open-source models are catching up with distillation, and all labs are sitting on debt and valuation that is impossible to fulfill even if they were alone in the market, so you essentially need either:

1) a breakthrough in performance/learning/model

2) regulatory capture to ensure open source models can be labelled as dangerous and banned so you can set the market rules yourself

Only one of the above is risk-free, and just a question of capital/lobbying rather than a "maybe".

They need to distance themselves from the increasing likelihood of legal repercussions after their systems have been found to commit what could reasonably be seen as felonies. That’s my guess at least.

Though for once I do actually agree with that specific leader, it’s incredibly annoying how anthropomorphic Claude is. Anthropic went way too far in that direction

I am so tired of this. Especially when MS, who has the worst LLMs of them all, is throwing the stones.

All of these companies need to be shut down.

Water is wet

More at 11