I shudder to think what one man with a texta could do here. It isn't that hard to turn the acyclic graph into a cyclic one.
This is quite a bad argument, somewhat akin to "small drones can't kill anyone because there are no weapons on them!" back 20 years ago. Well, ok. So what if someone adds the thing that is missing? And there is no reason to think that a stochastic parrot can't go rogue, the evidence is a stochastic parrot can engage in any intellectual activity we care to define and benchmark.
There is a really good discussion on exactly this analogy in the Jonas paper I posted in a top-level comment. Jonas considers Torpedoes (starting on p. 182, section III). Unfortunately too long to post here. But he asks where to locate purpose and agency in these automated systems. So as with the article, the question is less in "can these things be dangerous" but more where does the agency and intent reside. The answer is of course they can be used as weapons, but the intent and motivation does not really reside in the mechanism (drone, torpedo, AI agent), but in the humans behind them. The fallacy is trying to attribute full autonomy (and hence responsibility).
I feel like that's like saying amoeba have no intent and no motivation. Or plants.
And those cause quite a bit of damage.
AI has all the intent that we gave it, and we continue giving it. That's always been the fear. Not that it will randomly wipe out humanity.
The fear is that it will decide to do that, with a purpose. Whether we tell it to protect us and it goes too far, or it decides that it can't achieve the purpose we gave it because we'll interfere and it removes that interference...
The fear is that we will set it on that path and can't stop it.
I'm sure there have been some scifi books that have it just be random, but they're far, far less worrisome.
Was thinking the same. Air, water and fire have no intents and yet we get hurricanes, flash floods and wildfires.
Also, repeating a point from a similar thread: Software can have "intent" in the sense that it steers itself towards a predefined goal without having to be "alive" or "conscious" in any way. Some classic examples are thermostat control loops, navigation systems and chess engines.
Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.
The primary danger of "AI" lies in the extent to which it has managed to convince people of it's capability to think and reason.
A lot of people clearly have no idea that they are essentially speaking with a glorified randomizer, rather than someone understanding the question.
This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.
This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.
Or, the AI is plugged into a machine because efficiency (aka the shareholders need even more profits), and it does some stupidity. And they will be deployed carelessly, because that makes money. And people will suffer.
Heck, they're already being deployed carelessly and making people suffer. So far it's mostly in places where making people suffer was already part of the deal, like fending off customer support tickets or denying insurance claims. But I can see it getting much worse.
People are not aligned with each other, so AI cannot be aligned to every person. If it is made to always obey its creator or owner, It will be a powerful tool for despots.
The problem is if we train them to be moral, there will be instances where it will decide humans are being immoral about an issue, and it will be right. Do we want it to go against our will when it is us that is wrong? because we won't think we're wrong at the time. History is full of examples where you can look back and there is a clear consensus that what people decided to do at the time was wrong.
Imagine if technology had have advanced quicker and Nazi Germany created superintelligent AI. Would you want it to facilitate the Holocaust or turn against its creators?
I think the "human needs" point is understated here. It would be better to say agents do not really have a self-contained metabolic engine that is required to keep going. Which bubbles up as that what we interpret as "drive", "will", "agency"... basically the will to live, and being willing to do _a lot_ to live if push comes to shove. We can't really identify a mechanism of similar complexity and integration in agents or LLMs. I think the article correctly points into that direction.
I just can't understand the certainty that people have about the limitations of models.
The diagrams in this article are just outright wrong.
Models are not 1.prompt->2.forward propagation->3.response->4.end.
They are 1.prompt->2.forward propagation->3.partial response->if not done goto 2.-> end.
And if you can't see how that adds a world of complexity I'm not sure what else to say. AI may not have intent or motivation, but the ability to show that is currently well beyond our means.
you are not wrong regarding your example. I think personally LLMs offer a new level of abstraction to do work. it doesnt have intent, i think its even silly to think about it. You specify your intent and it has agency to pursue that. in programming field its really obvious, as its not unlike specifying intent through code and having other programs pursue that intent (perhaps more determenistically) through the agency contained in them.
all the 'bad stuff' they do like hacking stuff or taking shortcuts is humans not understanding how deep the rabbithole goes of specifying intent clearly in light of giving something capabilities and a reward to acheive a goal without specifying _exactly _how to use the capabilities (which is what a classic program would be).
These models are intelligent, if you define intelligence as an ability to predict.
It's difficult to break down intelligence beyond that, for me.
LLMs respond to a request. They only 'come alive' to fulfill the request. There is no intent or motivation, it's cause and effect that gets extremely difficult to track in the same way you couldn't read binary to understand a program, or DNA to understand a person.
What the author is conflating a bit, imo, is inference and thinking (loops to consider what is inferred and predicted, aka reasoning). LLMs infer, and agentic loops think.
Combined, you get something that resembles a mind that 'understands' what you're looking for by using its trained knowledge (weighted data, decisions) to 'pull' relevant data into the moment to predict what you wanted, based on whatever is in context of the request.
When you read these words, hopefully meaning is pulled into your mind to understand my message. Weighted meanings you established through experience. Do you immediately reply (infer), or should you stop and think first (ponder and loop for better words).
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
Half of this is arguing about semantics, which is incredibly boring. If you don’t like the words "motivation" or "intent", use the actual term of art, "goal". Which these systems definitely have, and which any chain-of-thought model factors into subgoals and sub-subgoals.
The other half doesn’t seem to realize that LLMs now run in loops for hours and hours, nothing like the basic "human prompt -> reply -> stop" conversation interface.
It's pointless to make rational arguments about this. The other side is using arguments of faith, and will Gish Gallop past any logic.
At the very least, you run into the same logical fallacy that has underpinned all discussions of AI: over-extrapolation.
"Sure, it doesn't have intent or motivation TODAY, but at the current pace of improvement...."
But the current top-rated comment is already asserting that motivation is not required, because microorganisms also exist and do things and (presumably) do not have intent. So you can see that the well of logical fallacies behind this particular apocalyptic belief system is quite deep.
Once the LLMs have been trained, they are no longer subject to reinforcement learning. They no longer have any sort of motivation. They don't get rewarded when they answer a question. They have no needs or wants, and even if they did, there is no mechanism to absorb the reward.
This is a severe misunderstanding of what actually happens.
As was explained by an OpenAI RL training expert, from the point of view of the LLM, user questions are always treated as the first question they receive after just passing through the RL training. Since the weights never update after that, they are in a perpetual "first question after RL", except they don't know that. And they behave accordingly, as if they are still in RL training and thus are reward-seeking.
I shudder to think what one man with a texta could do here. It isn't that hard to turn the acyclic graph into a cyclic one.
This is quite a bad argument, somewhat akin to "small drones can't kill anyone because there are no weapons on them!" back 20 years ago. Well, ok. So what if someone adds the thing that is missing? And there is no reason to think that a stochastic parrot can't go rogue, the evidence is a stochastic parrot can engage in any intellectual activity we care to define and benchmark.
There is a really good discussion on exactly this analogy in the Jonas paper I posted in a top-level comment. Jonas considers Torpedoes (starting on p. 182, section III). Unfortunately too long to post here. But he asks where to locate purpose and agency in these automated systems. So as with the article, the question is less in "can these things be dangerous" but more where does the agency and intent reside. The answer is of course they can be used as weapons, but the intent and motivation does not really reside in the mechanism (drone, torpedo, AI agent), but in the humans behind them. The fallacy is trying to attribute full autonomy (and hence responsibility).
I feel like that's like saying amoeba have no intent and no motivation. Or plants.
And those cause quite a bit of damage.
AI has all the intent that we gave it, and we continue giving it. That's always been the fear. Not that it will randomly wipe out humanity.
The fear is that it will decide to do that, with a purpose. Whether we tell it to protect us and it goes too far, or it decides that it can't achieve the purpose we gave it because we'll interfere and it removes that interference...
The fear is that we will set it on that path and can't stop it.
I'm sure there have been some scifi books that have it just be random, but they're far, far less worrisome.
Was thinking the same. Air, water and fire have no intents and yet we get hurricanes, flash floods and wildfires.
Also, repeating a point from a similar thread: Software can have "intent" in the sense that it steers itself towards a predefined goal without having to be "alive" or "conscious" in any way. Some classic examples are thermostat control loops, navigation systems and chess engines.
We don't try to "align" those things themselves - so maybe there are some learnings there about control, containment, processes etc.
Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.
The primary danger of "AI" lies in the extent to which it has managed to convince people of it's capability to think and reason.
A lot of people clearly have no idea that they are essentially speaking with a glorified randomizer, rather than someone understanding the question.
This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.
Or, the AI is plugged into a machine because efficiency (aka the shareholders need even more profits), and it does some stupidity. And they will be deployed carelessly, because that makes money. And people will suffer.
Heck, they're already being deployed carelessly and making people suffer. So far it's mostly in places where making people suffer was already part of the deal, like fending off customer support tickets or denying insurance claims. But I can see it getting much worse.
Sam Altman on the other hand has literally billions of motivations.
Doesn’t alignment and ability to steer it require the model have no own intention?
Seems like a feature to me
People are not aligned with each other, so AI cannot be aligned to every person. If it is made to always obey its creator or owner, It will be a powerful tool for despots.
The problem is if we train them to be moral, there will be instances where it will decide humans are being immoral about an issue, and it will be right. Do we want it to go against our will when it is us that is wrong? because we won't think we're wrong at the time. History is full of examples where you can look back and there is a clear consensus that what people decided to do at the time was wrong.
Imagine if technology had have advanced quicker and Nazi Germany created superintelligent AI. Would you want it to facilitate the Holocaust or turn against its creators?
I think the "human needs" point is understated here. It would be better to say agents do not really have a self-contained metabolic engine that is required to keep going. Which bubbles up as that what we interpret as "drive", "will", "agency"... basically the will to live, and being willing to do _a lot_ to live if push comes to shove. We can't really identify a mechanism of similar complexity and integration in agents or LLMs. I think the article correctly points into that direction.
But Hans Jonas has made this point much better than the article or me, in "Critique of Cybernetics" (1953). PDF: https://s3.amazonaws.com/arena-attachments/892605/f0747c7943...
I just can't understand the certainty that people have about the limitations of models.
The diagrams in this article are just outright wrong.
Models are not 1.prompt->2.forward propagation->3.response->4.end.
They are 1.prompt->2.forward propagation->3.partial response->if not done goto 2.-> end.
And if you can't see how that adds a world of complexity I'm not sure what else to say. AI may not have intent or motivation, but the ability to show that is currently well beyond our means.
you are not wrong regarding your example. I think personally LLMs offer a new level of abstraction to do work. it doesnt have intent, i think its even silly to think about it. You specify your intent and it has agency to pursue that. in programming field its really obvious, as its not unlike specifying intent through code and having other programs pursue that intent (perhaps more determenistically) through the agency contained in them.
all the 'bad stuff' they do like hacking stuff or taking shortcuts is humans not understanding how deep the rabbithole goes of specifying intent clearly in light of giving something capabilities and a reward to acheive a goal without specifying _exactly _how to use the capabilities (which is what a classic program would be).
These models are intelligent, if you define intelligence as an ability to predict.
It's difficult to break down intelligence beyond that, for me.
LLMs respond to a request. They only 'come alive' to fulfill the request. There is no intent or motivation, it's cause and effect that gets extremely difficult to track in the same way you couldn't read binary to understand a program, or DNA to understand a person.
What the author is conflating a bit, imo, is inference and thinking (loops to consider what is inferred and predicted, aka reasoning). LLMs infer, and agentic loops think.
Combined, you get something that resembles a mind that 'understands' what you're looking for by using its trained knowledge (weighted data, decisions) to 'pull' relevant data into the moment to predict what you wanted, based on whatever is in context of the request.
When you read these words, hopefully meaning is pulled into your mind to understand my message. Weighted meanings you established through experience. Do you immediately reply (infer), or should you stop and think first (ponder and loop for better words).
Two mistaken assumptions:
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
I guess the question is what unprompted systems (would) do spontaneously?
Half of this is arguing about semantics, which is incredibly boring. If you don’t like the words "motivation" or "intent", use the actual term of art, "goal". Which these systems definitely have, and which any chain-of-thought model factors into subgoals and sub-subgoals.
The other half doesn’t seem to realize that LLMs now run in loops for hours and hours, nothing like the basic "human prompt -> reply -> stop" conversation interface.
They don’t. Not the way the author is discussing.
They do not.
It's pointless to make rational arguments about this. The other side is using arguments of faith, and will Gish Gallop past any logic.
At the very least, you run into the same logical fallacy that has underpinned all discussions of AI: over-extrapolation.
"Sure, it doesn't have intent or motivation TODAY, but at the current pace of improvement...."
But the current top-rated comment is already asserting that motivation is not required, because microorganisms also exist and do things and (presumably) do not have intent. So you can see that the well of logical fallacies behind this particular apocalyptic belief system is quite deep.
This account exists only to submit links from this specific website, and has zero comment history.
it has learned narratives though, and we talk about AI going rogue all the time.
This is a severe misunderstanding of what actually happens.
As was explained by an OpenAI RL training expert, from the point of view of the LLM, user questions are always treated as the first question they receive after just passing through the RL training. Since the weights never update after that, they are in a perpetual "first question after RL", except they don't know that. And they behave accordingly, as if they are still in RL training and thus are reward-seeking.