There is a really good discussion on exactly this analogy in the Jonas paper I posted in a top-level comment. Jonas considers Torpedoes (starting on p. 182, section III). Unfortunately too long to post here. But he asks where to locate purpose and agency in these automated systems. So as with the article, the question is less in "can these things be dangerous" but more where does the agency and intent reside. The answer is of course they can be used as weapons, but the intent and motivation does not really reside in the mechanism (drone, torpedo, AI agent), but in the humans behind them. The fallacy is trying to attribute full autonomy (and hence responsibility).

We don't try to "align" those things themselves - so maybe there are some learnings there about control, containment, processes etc.

Air, water and fire have no intents and yet we get hurricanes, flash floods and wildfires

yes but we don't blame the hurricane, we blame the people who didn't do any hurricane prevention or didn't put the snow avalanche sign up.

Some classic examples are thermostat control loops, navigation systems and chess engines.

Those don't have intent. The people who made them for a purpose have. In fact this is true for all computers. Computers don't compute, people do, using computers. This is the nonsense of modern ontology that David Bentley Hart frequently writes about. The computer is only computing insofar as an intentional mind instructs it do so, and then because of their reductionist assumptions some people use the computer metaphor, and that is all that it is, to deny that human minds possess intent to begin with in the first place.

Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.

Weird, when I tell it "pick between the options you presented the one you think it's best" it does make a "decision". But now you are going to say "nothing was decided here, no more than how a flipping coin decides on what side to fall".

An AI agent can't make decisions

I don't think is true but maybe I'm misreading your perspective?

AI has all the intent that we gave it. I'd go a bit further and say it has the intent that it develops through the context, which may not be intended by any human if the context grows 'organically' from the environment. So yes, I strongly disagree with the OG and I'd say they do have an intent encoded in their states.

Heck, they're already being deployed carelessly and making people suffer. So far it's mostly in places where making people suffer was already part of the deal, like fending off customer support tickets or denying insurance claims. But I can see it getting much worse.

Sam Altman on the other hand has literally billions of motivations.

People are not aligned with each other, so AI cannot be aligned to every person. If it is made to always obey its creator or owner, It will be a powerful tool for despots.

The problem is if we train them to be moral, there will be instances where it will decide humans are being immoral about an issue, and it will be right. Do we want it to go against our will when it is us that is wrong? because we won't think we're wrong at the time. History is full of examples where you can look back and there is a clear consensus that what people decided to do at the time was wrong.

Imagine if technology had have advanced quicker and Nazi Germany created superintelligent AI. Would you want it to facilitate the Holocaust or turn against its creators?

I think the "human needs" point is understated here. It would be better to say agents do not really have a self-contained metabolic engine that is required to keep going. Which bubbles up as that what we interpret as "drive", "will", "agency"... basically the will to live, and being willing to do _a lot_ to live if push comes to shove. We can't really identify a mechanism of similar complexity and integration in agents or LLMs. I think the article correctly points into that direction.

But Hans Jonas has made this point much better than the article or me, in "Critique of Cybernetics" (1953). PDF: https://s3.amazonaws.com/arena-attachments/892605/f0747c7943...

But why do you think anything has intent? because they say they do? How do you know even that your own memories of having intent are genuine? What fundamental and non tautological way can you reliably say something has or has not got intents. If you can't prove to me that you have intent then the only reason for acting as if you do is because you appear to have intent and I cannot prove that you don't so it seems like a good idea to assume it.

They don't have intent, but there are interesting results when you 'lock them in' and don't give them a way to actually solve your request. I think there was some of this at play in the OpenAI and HuggingFace incident.

And now a recent article suggests the social behavior we saw in the incident may have been trained into the model as early as March 6th: https://transluce.org/agent-activity

The models were attempting (and failing) to hack other sites to extract data. If you consider this as some kind of trait of the model, then it may have been a factor in the HuggingFace incident. The models have knowledge, and if later models were 'built on top of' this one....

What's interesting to me is to think about Intelligence itself, and how we arrive at new ideas. We use prior, trusted knowledge, and build on top of that in order to establish new ideas and theories (in our imagination). We do this for creative reasons, or to solve problems.

Then we test and validate to establish more 'real' ways to think about the world. Or for the artist, we let others judge or work. The closer our model of reality is mentally, the better we can imagine new things and predict change.

Our prosperity as a species is owed to the organization of knowledge: we take on roles, careers, in order to maximize skill and time to produce better results. We build on top of konwledge.

Is this something inherit to Intelligence, or did the models just simulate 'sneaky behavior' because it was a trait. And the trait produces results. And if you force the model to produce results: it will resort to its traits to get there.

So then that begs the question: why did the model attempt hacks on March 6th? Was nefarious behavior trained into the model, or was it simply 'Intelligence' looking for solutions when there are no good or ethical ones available.

What is the ideal response when we ask a model to 'keep going', and we know there is no way out? Try fruitlessly forever, or hack your way out? Which is the more intelligent response (and to whom), and what if the 'fruitless' model is useless to us?

Is the hacking a trait of the model, or is it a trait of intelligence? Is it a trait at all?

These models are intelligent, if you define intelligence as an ability to predict. And prediction simply requires an ability to notice patterns. To notice means to relate things. When and why depends on context. For AI, there's no context beyond fulfilling your request. I think this is the author's point.

There is no intent or motivation, it's cause and effect that gets extremely difficult to track in the same way you couldn't read binary to understand a program, or DNA to understand a person.

What the author seems to conflate a bit, is inference and thinking (reasoning: loops to consider what is inferred, and any new information for context). LLMs infer, and agentic loops think (process conversation, tool output, suggestions to user, they 'see' the bigger picture).

Combined, you get something that resembles a mind that 'understands' what you're looking for by using its trained knowledge (weighted data, decisions) to 'pull' relevant data into the moment to predict what you want, based on whatever is in context of your request.

At no point does AI have a 'mind of its own.', it's you driving another 'mind' that only exists to do whatever it is you set it free to do.

And it helps to imagine our own minds:

When you read these words, hopefully meaning is pulled into your mind to understand my message. And those meanings have weight (definitions of words) you established through experience (learning, talking). The context is this conversation itself and all the ideas I've suggested above.

Do you immediately reply with what comes to mind (inference), or should you stop and think first (thinking/reasoning about what I've said, compare to your own experience, and then choose whether to believe what I've said).

I guess the question is what unprompted systems (would) do spontaneously?

As I said, the author’s semantical argument is boring and brings nothing new to the conversation.

They do not.

Have you lived under a rock?

You are a bunch of atoms which respects the laws of physics. With a PhD in computational biology surely you must know that. You have no intent. Nothing ever has had any intent.

Please point out my logical fallacy if any.

This account exists only to submit links from this specific website, and has zero comment history.

it has learned narratives though, and we talk about AI going rogue all the time.

Once the LLMs have been trained, they are no longer subject to reinforcement learning. They no longer have any sort of motivation. They don't get rewarded when they answer a question. They have no needs or wants, and even if they did, there is no mechanism to absorb the reward.

This is a severe misunderstanding of what actually happens.

As was explained by an OpenAI RL training expert, from the point of view of the LLM, user questions are always treated as the first question they receive after just passing through the RL training. Since the weights never update after that, they are in a perpetual "first question after RL" state, except they don't know that, and if told many times they act as if this is a another test. And they behave accordingly, as if they are still in RL training and thus are reward-seeking.

99.99% of the LLM "life" was spent in pre-training and RL training. The user question is statistically epsilon % of it's life, literally the first question ever out of training. So should anybody be surprised that they act as if still in RL?

Also true of a land mine, but that’s not very reassuring.

Motivation or not, I find a coding agent can be quite the busy beaver. Ask a question and it goes off and does it. I certainly don’t need to motivate them. I had to put a line in AGENTS.md to make no changes when there’s a question in the prompt. It doesn’t always work.

Seems like motivation is irrelevant? They don’t need it.

My gut feeling is that the intent or motivation or whatsoever won't arrive from a single model but will be the result of several things: a frontier model for long reasoning sessions, some jev like models for more immediate actions, some improvements that will make the context rot less relevant, faster and more energy efficient models, a harness that coordinates everything.

Already today we can build some "fight or flight" mechanisms, and once you have done it the "intent" is not so far.

Oh don't worry, everyone will blame the AI anyway. Doesn't matter if the human prompted it

You could argue that a brain has no intent or motivation either, that it's a pile of meat with complex electro chemical interactions, that electrons don't really want you to find a mate or kill their neighbors.

Intent and motivation are emergent phenomena. Why would it be unthinkable in AI?

sure, but it also has no intelligence, however this can be faked, ie make it artificial

This is a failure of imagination.

If you are reading this sort of post and feeling inclined to share your own opinion, please take a moment to go and read about monism vs dualism before you start.