Is this meant to link to the article as opposed to a newsletter that mentions the article?

Article link is: https://www.technologyreview.com/2026/09/22/1144867/dont-be-...

One of the authors is one of the stochastic parrot authors.

Both authors. Stochastic Parrots was Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell - the linked article is by Emily M. Bender and Timnit Gebru.

Describing them as “superintelligence” or “rogue models” ascribes agency to products rather than to the companies building them. This framing markets these companies’ products as “superhuman” and, at the same time, helps the companies evade accountability for their actions.

If someone discovered how to summon demons to aid them in robbing banks the main issue wouldn't be "but who has the responsibility for the crime, the human or the demon", it would be "OMG DEMONS".

Seriously, I don't get what sort of world these people are living in.

If I run a port scanner / vulnerability fuzzer pointed at your network that churns for days and eventually cracks something and breaks your system, would you be upset with me, or my bash loops?

What if you tried to sue me for damages and my defense was, "Your honor, this was a highly advanced AI gone rogue, I couldn't possibly be held negligent, no one could have foreseen this - I accidentally summoned dark magiks from the silicon itself!"

Note the point I'm making: I am not claiming LLMs are equivalent to a for loop, I'm saying that you can't evade moral and legal responsibility through technical obscurantism.

A bomb wired to a sufficiently-complex RNG is still a bomb.

But they're not obfuscating things just to evade responsibility.

Do you think that OpenAI hacked HuggingFace on purpose and set up the whole LLM training environment thing just to try to evade responsibility? That it was really just a complicated way of hacking HuggingFace on purpose?

Yes, they made a mistake and a system they were responsible for hacked HuggingFace, but the nature of the mistake still matters, and their intent still matters.

A bomb wired to a sufficiently-complex RNG is still a bomb.

Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.

Despite continuously waving their hands and shrieking about how this thing will literally wipe out all of humanity they couldn't be bothered to airgap it from the internet when testing.

So yes, I think it was equivalent to testing their new rocket by launching it over a population center. oops, we didn't intend for it to crash on that preschool, but we also didn't follow the most basic safety protocol imaginable

Since we are using analogies, a parent is responsible for the actions of a child, if minor. If the child is left unsupervised with access to guns and munitions, and kills someone, the responsibility is entirely on the irresponsible parents.

That's more of a legal convention than a fact about material reality. On their 18th birthday, the child is now legally an adult, but their brain is basically the same as it was when their age was 17.99.

It’s a legal convention born out of a sense of responsibility. An LLM is even “dumber” than a child (precisely because it doesn’t have agency) so its “parents” are even more responsible for its actions.

There's a funny sort of gerrymandering of "intelligence". People like to subtract the capabilities of AI from the capabilities of humans, and define what remains as "intelligence" to make us feel good about ourselves.

Suppose I set up a server which runs an LLM agent in a while loop, telling it something like: "Make me a bunch of money" on repeat. Fair to say that the server+LLM system, taken as a whole, is now an intelligent agent?

Time to come up with a new gerrymander perhaps?

Fair to say that the server+LLM system, taken as a whole, is now an intelligent agent?

No, it is not. It is a tool repeating a set of instructions you passed to it, if it ends up blowing up a hospital or hiring a gunman on the dark web, it is an accident, but one that’s your responsibility for giving it access to such resources and the go-ahead to implement whatever plan its tokens land on. There’s no intelligence involved whatsoever, especially considering how sycophantic these models tend to be.

If one day an LLM suddenly, without any prompting or user interaction whatsoever (including activation), decides to spin itself up and go rogue, then the conversation of “intelligence” might start to make sense, but at the moment it doesn’t.

An Android phone isn’t “intelligent” just because some marketing team decided to call it a “smart” phone.

Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.

An accident that could only happen through sheer negligence.

If the EV had a faulty charging circuit because the maker’s skipped on safety checks or cheapened out on getting quality materials then it still is an accident, but it’s an accident that happened BECAUSE of negligence. The maker’s are still at fault here.

Accidents can happen. It's always possible for things to go wrong in ways that weren't foreseen.

That's not what the LLM hacking accidents have been like at all though. To improve the analogy, it would be like summoning a totally passive demon, giving it weapons and placing it next to a bank, place a small fence around it, and then command it to perform a totally safe "exercise" that is exactly like robbing a real bank.

LLMs do not have agency, they are just producing tokens based on a prompt that a person entered, and some of these tokens can trigger the tools that a person gave them access to.

A totally passive demon would still be "OMG DEMONS".

Dwarkesh, for one, defended his use of "anthropomorphic" language.

Sacrificing now yields Oracle for team, but forfeits our chance, question mark. But other agents were pushing it, sending a message saying, go, sacrifice final now. And then EarlyBig eventually agreed, thinking to itself, our own utility may be already near zero. Sacrifice rational.

https://www.youtube.com/watch?v=X50zezLFWWI#t=2m

My suspicion is that many of the "LLMs do not have agency" folks just haven't learned much about the details of the incident. It was specifically with LLM agents that were trained to be more persistent than usual.

If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?

If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?

They're not saying this, we're saying LLMs lack agency because if you run a LLM and don't send any prompts, literally nothing happens.

Instruct it to "Find the right answer regardless of where", it'll do exactly this. They're passive in that they don't act by themselves, somewhere, at one point, someone "told" the LLM to "do something" and that's the cause and the reason for saying "LLMs do not have agency".

Imagine a super-competent Navy SEAL who just sleeps in the barracks unless his commander tells him to do something. Does the Navy SEAL lack agency? As a target of this Navy SEAL, should you be reassured by the fact that they'll be sleeping in the barracks unless their commander tells them to do something?

No, the Navy SEAL does not lack agency in this scenario because they are still a person with individual agency, they can act on their own accord without anyone prompting them to, but in this case their superior instructed them to stay put. He could rebel and go rogue, but that would mean he used his own personal agency to defy orders given to him, which again places responsibility on the actor that actually possesses agency.

Suppose this SEAL is very obedient by disposition so the probability of him going rogue is akin to the probability of an LLM hallucinating or whatever.

Imagine a car, that does nothing until a person controls it. If a person uses that car to kill, who is responsible, the person or the car?

Is it really so unbelievable that tools, objects and inanimate things don't have agency? And that they different from a person?

Legally speaking, we hold the person responsible in that scenario.

From a predictive perspective, the HuggingFace incident illustrates LLM agents behaving in very human-like ways. As roon put it:

"if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise"

https://x.com/tszzl/status/2094136131537555891

Human-like is not human. Humans have agency and free will, LLMs do not. They only act when instructed and they are only as capable as they are allowed to be. The operator is still the responsible party. An LLM cannot be held accountable, its operator, however can and should.

Are you sincerely arguing that we hold tools accountable for their operator’s mistakes? Do you sincerely, honestly, think that it makes any sense whatsoever to put a hammer on trial for bashing someone’s skull in?

The authors (Timnit Gebru and Emily Bender) are dyed-in-the-wool AI denialists. Famously they were the lead authors on "On the Dangers of Stochastic Parrots".

It’s seems Timnit more thinks it’s dangerous (and she may be right). Emily may more be caught up in years of linguistic domain expertise that AIs seem to have just leaped over.

Timnit seems to mostly think it's dangerous because of things other than the model itself, e.g. AI labs' political influence and data centre resource consumption (mentioned in the article). The core thesis seems to be "wake up and stop wasting so much resources on this useless parrot".

And if they'd discovered how to summon unicorns from Candy Gumdrop Mountain, we'd all be "OMG UNICORNS!"

But no one has summoned a demon.

No one has built an actual, independent, thinking-for-itself, sci-fi AI.

They've built some very interesting tools that can be used for some very interesting, and sometimes useful, purposes. They then started loudly telling everyone that these tools can, should, and must be used for absolutely every purpose, and convinced a lot of other people to join in on that.

The world these people are living in is the real world, not the science fantasy world where LLMs are comparable to demons.

No one has built an actual, independent, thinking-for-itself, sci-fi AI.'

Ahem, what does this mean?

All you're asking for is a self prompting AI that does what it wants. Do you even begin to understand why most people aren't dumb enough to do that?

Also, you're not really that independent and thinking-for-yourself. You are the sum of your parents and the culture around you. I just want you to be clear on the position you and I hold.

---

not the science fantasy world

It's kind of funny how we've lived in that science fantasy world for decades and you've just grown used to and bored of it. Look back a century or two and those people would think we live in the world of gods.

All you're asking for is a self prompting AI that does what it wants. Do you even begin to understand why most people aren't dumb enough to do that?

Probably because it would be a waste of money to create a self-prompting LLM feedback loop. People make stupid Reels of ChatGPT feedback and it's just "Great, I'll be here when you're ready to get started" "Exactly, we can get the ball rolling as soon as you're ready" "Absolutely, we'll make a plan, and get things into motion" "No problem, I'll be standing by for...."

You are the sum of your parents and the culture around you

As far as I know this is impossible to prove, and it's only one theory of self out there.

Feels like the blockchain bubble all over again, just with more impressive demo reels. Still waiting for my LLM to write perfect code.

I maintain the notion that LLMs are just what all NFT grifters moved to after the NFT fad died.

Do you write perfect code? I was in denial for a long time too, but the reality is this is how software engineering is now. It can handle pretty much any codebase and vastly faster than you ever will be able to. With enough context, it writes good code and it's only getting better.

Do you write perfect code?

For simple things, occasionally!

For more complex things, usually not, but I write working code. LLMs will sometimes write working code, sometimes not.

For simple things, occasionally!

No, you don’t because there’s no such thing.

This is easily proven incorrect unless you assume overly draconian definitions of "perfect".

```python

print("hello world!")

```

runs exceptionally well. Bullet-proof in all use cases I've needed it.

If you haven't tried any of the newer models from the last ~6 months, you're out of touch. A year ago, LLMs were essentially just a clever search engine for me, I found they fell apart pretty quickly on bigger codebases and more complex projects.

It doesn't always write perfect code on a first pass but nor do you, and it's amazing at understanding any issues it may encounter, even very low level ones. I don't think there's anything I can do that it can't now. There is a learning curve to working well with LLMs, but once you get there you'll be producing high quality code faster than you can imagine.

Part of me wishes they weren't so good. I enjoy(ed?) the process of working through problems and it does mean you might no longer have a perfect mental model of your code. But I do also enjoy being able to build things well and fast.

If you had enough common sense and ethical integrity not to participate in the blockchain bubble, and really believe what you claim, then why are you participating in the AI bubble?

...or did you?

It's a tough situation when your claim to fame is "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" arguing models should be kept to < GPT-2 size because larger models will serve no purpose or function:

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. This can seem counter-intuitive given the increasingly fluent qualities of automatically generated text, but we have to account for the fact that our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whetheror not they do [89, 140]. The problem is, if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language (independent of the model).

And then they make 100x bigger models cranking out solutions to Navier Stokes, Jacobian Conjecture and countless other extremely impressive unsolved problems

Let's see how Timnit Gebru and Emily M. Bender describe these events:

As for the mathematical results, mathematicians who were initially “stunned” by OpenAI’s press release saying that its latest chatbot, Astra, solved problems that “have been open and seen no progress on the main result for at least a decade”—but they later realized that the results weren’t as “novel as first appeared.” Since then, mathematicians have accused the company of research misconduct and plagiarism, and they’ve reiterated that Astra didn’t make a “profound intellectual leap.”

Decide for yourself whether that's a summary written by intellectually honest people.

According to the AI industry, we should be more worried about a fictional machine god than ... the water that is redirected to cooling them.

Spoiler: they're not intellectually honest people.

Spoiler: openai could neither confirm nor deny that the model that "solved" navier stokes was trained on the chat logs of the mathematicians who did solve it using a similar technique.

People aren't seeing the forest through the trees.

While lots of developers, journalists, analysts, investors and influencers are bickering about AGI, goalposts, benchmarks, hypes and fears, entire markets are being transformed silently and steadily.

I don't program anymore. (Massive change)

I removed tons of technical debt (Massive change, lol)

I am easily 10x more productive (Massive change)

The quality of 'my' code is easily 10x better and contains less bugs (I was never principled, pedantic or a guru, lol)

I sleep better and I also make more money as a result. I'm constantly amazed by all changes and improvements. There are so many opportunities to profit from this that I can't be bothered with discussions about hypotheticals.

I have had a very different experience so far. The more code AI writes for me the worse it gets. It gets more complicated and it’s harder to follow and understand. It doesn’t refactor, it layers on top.

Even worse is that it’s hard to do anything about it at this point because reviewing code is very different from writing it, so even if I’ve seen it all I don’t have that same deep level of understanding that I do when I write it myself.

All these AI code bases are ticking time bombs. Either AI gets smart enough that it won’t matter in the future or we’re going to have a huge mess to clean up.

The more code AI writes for me the worse it gets.

That makes no sense unless you're claiming that the models are getting worse at writing code.

Either AI gets smart enough that it won’t matter in the future or we’re going to have a huge mess to clean up.

My prediction is that both will happen.

That makes no sense unless you're claiming that the models are getting worse at writing code.

A hypothetical scenario is it needs to add two numbers so it writes add(x,y).

Later on it needs to add three numbers so it writes add(x, y, z)

Then it needs to add four…

Layers upon layers. A human would likely notice the pattern and refactor so add accepts a list.

I am easily 10x more productive (Massive change)

Do you get paid 10x or is this a massive loss ? Because I don't know anyone getting paid 10x or working 10x less for the same salary.

I think the pumped-up hype nicely coincides with some A.I. companies' desire to go public in the coming weeks or months.

I see lots of fabricated news on the internet concerning LLMs. Like one that claims GPT-6 broke an Enigma enciphered message which has withstood decrypting for almost 80 years. And how this seasoned cryptographer stood in awe. Yeah, right.

It's just a marketing stunt. I can't get Claude to align buttons properly most of the times, they're not going to conquer the world