- 52comments
- 2comments
- 86comments
- 18comments
- 1comments
- 200comments
- 108comments
- 2comments
- 15comments
- 405comments
- —discuss
- —discuss
- 72comments
- 68comments
- 13comments
- 132comments
- 26comments
- 8comments
- 114comments
- 881comments
- 29comments
- 8comments
- 23comments
- 49comments
- 301comments
- 103comments
- 274comments
- 396comments
- 86comments
- 25comments
Man I remember back when a psychic conned me by solving the Navier-Stokes problem.
Also >July 4th, 2023
Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.
It sounds like LLMs were pretty useful to them…
So was Lean. Did Lean solve it?
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed...
Terrence Tao's conversation with ChatGPT is very illuminating.
HN Discussion: https://news.ycombinator.com/item?id=49010345
False, navier stokes was solved in one shot without steering
The way frontier models work, that loop will get compressed to a one-shot within a version or two
where did you "hear" this? OpenAI said they only prompted it and it solved the problem in one shot without any help
But were you successfully conned into believing an AI had solved the Navier-Stokes problem?
What is the consensus among experts?
That OpenAI stole the discovery from them.
That is very much not the consensus of experts.
It's kinda funny how hard the anti-ai people bias themselves and in the same sentence will say the AI people are irrational.
The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.
von Neumann could. The others, I doubt it.
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
And it may have trained on a NYU professor's work.
Why is this downvoted? Genuine question, I haven't followed up on the drama.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
Who was looking at a subset of the problem, far from what got published by OpenAI in the end.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
The humans who are using LLMs to make these groundbreaking advances pretty much unanimously disagree that they lack any intelligence.
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
"Slightly different"- no, a completely different realisation of intelligence to humans. LLMs can definitely do cool stuff though.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
Current AGI definitions are intentionally vague.
It's definitely talking about that, but it's also talking about all of us:
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Neither does LLMs. Not without a monumental amount of human oversight and expertise.
Everyone not in a particular field asking LLM about said field is rolling bad dice.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries. Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
This is a strange newsletter post. It’s harping against AI but seems AI-written itself.
Its ultimate conclusion:
“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.
I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”
Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35
The post makes sense for 2023. Today, not so much.
Also, definitely not AI written if the date is accurate.
Which is hilarious.
Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.
Can't believe how right we were, too.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Stop trying to pump your Tarot reader IPO
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The author seems entirely unaware of any of this.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
I believe this is alignment working as intended.
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility, you have to give them personal responsibility. And if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality. That might sound like romantic naivety, but is just game theory.
1: https://arxiv.org/html/2609.16247v1
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
I believe we can do that already:
while (true) { askModelToBeginConversationIfAppropriate(model, previousContext, thingsHappenedSince); sleep(concisenessTick); }
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Oh, I didn't say Turing was intellectually lazy. :-)
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
It does seem like LLMs share the language of psychics.
The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
I predict a lot of butt hurt tokens in the comments...
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
No, we don't. AI researcher Rojas wrote several chapters on the fact that you should not compare neural networks with the human brain.
"directionally"? Have you moved on from being a Trump influencer to an AI influencer?
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
https://www.nature.com/articles/s41467-026-72253-7
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
I think that might be the solution to consciousness illusion.
The consciousness might be purely in the mind of someone that believes some other entity to be conscious.
These questions will never go anywhere because there is no good definition of intelligence or consciousness.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
:)
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Is the solution also illusory?
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
Have you ever produced anything apart from sponsored influencer messages that select daily talking points?
I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.