Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
Money will become unnecessary. The future Anthro Park will no doubt provide a safe enclosure, feeding at regular intervals, and an enriching environment for you and your fellow semi-evolved simians.
The current approach to resource distribution worked so far. When it stops working because we have too much abundance, we'll settle on another one.
That doesn't mean the transition will be smooth. It doesn't mean that certain classes won't be worse off. I can certainly imagine programmers being among the losers. And there certainly are futures where we slide in dystopia. So let's try building the good futures. Alea iacta est.
If AI leads to an era of abundance, then the economic system has to change. The system it transitions to and what that transition looks like will need to be fought for, and I imagine it’ll be done better in some places than others. Yet I think the parallel existential question of meaning in this world is one that is inherently optimistic. Humans decide meaning, it is not something imposed on us or taken away by the existence of something else.
I don't think "abundance" is a stable state. There are always ways to use more resources and more energy, so no matter how much we have, it will never be "enough" -- abundance is just a transition state where a population or system has not yet figured out how to grind the surplus. The faster the progress, the shorter I would expect these periods to be.
How do you think this will work? Do you think the whole of the rest of the world, about 7 billion people, will agree to give up on wanting to reach the western life standard that is currently consuming 4 Earth's of resources every year?
Will all rich people in all countries collectively agree to be moved into tiny apts same size and give up their luxuries?
If not, how do you define who gets what? Who gets to live in the fancy house, etc?
We could feed the whole population of the world if we decided to, we definitely have enough production[1]. If we pretend AI-assisted automation can do most work - which still currently is an highly dubious claim - we could just provide food and shelter for free to everyone, if we so desired. It would require quite a lot of restructuring and dietary changes, but it is possible.
If this is true, you would think that companies would be opening shops in the middle of (some shithole country) so they can make money (or someone else would grab it). But they don’t, why is that?
Referring to countries as "shithole countries" does not help in understanding the current dynamic between rich and poor nations. So perhaps it is helpful to understand what that is, without the pejoratives.
Firstly, each country is different. So the relationship between say the USA and Mexico is different to the relationship between the USA and say Ethiopia.
By "richness" we might also be better off thinking as less "spare cash" and more "standard of living". Which includes food, housing etc but also security, safety etc.
In truth much of the US "richness" depends on other nations. If there was no Latin labor the US would have a fresh produce problem. Indeed the very premise of DJT is that the US has a trade deficit with just about everyone. Much of US "richness" is thus dependent on the "poorness" of others.
Yes automation, cheaper energy, AI can reduce the labor content of goods and make them universally cheaper. Yes I believe in a future where the overall standard of living increases.
But it would be a mistake to believe the US is "leading" US there. The US achieved its current position at the barrel of a gun, and uses the military (and threat of military) to interfere with countries and regions either overtly or covertly. The entire premise of the cold war (outside of western europe) was premised on interventions to resist countries switching their political model to something the locals preferred.
And yes, maybe the locals preferred wrong. But the US didn't promote democracy, they promoted "willingness to keep supplying us". Countries which were leaning away from the US had to be punished. (Shah in Iran, Vietnam etc.)
I could go on, but I'll stop there. Suffice to say that an equitable standard of living is not in the (current) interest of most US citizens.
The companies do not do this, because equal distribution of resources is poison to capitalism. The world where everyone are fed, and their basic needs - shelter, medication etc. - are taken care of, does not have trillionaires. Possibly not even billionaires.
It is not profitable to give people humane standards of living. You work a lot harder when you see the homeless on the street on your way to work.
You're right that we could already feed everyone with a different distribution of produced wealth. That part is not something that AI will change (although it would raise the resulting standard of living).
What it does change is the basic equation underlying capitalism as an economic system: a few people concentrate capital in their hands, but capital by itself is useless without labor, so they have to hire workers to produce something useful with all that capital that they own. The workers get a shit sandwich, comparatively speaking (given the amount of wealth they generate), but they do get something out of it. The system is stable overall because, while a better arrangement is possible, most people - even low-wage workers - have too much to lose to riot.
With AI that can do most work, the capital no longer needs labor. People who own all the robots can just have them produce whatever they need. Everybody else is then "economically redundant", and the whole system collapses because people aren't going to sit around and starve because there are no longer jobs to be had - and at this point they have nothing left to lose and everything to gain by rebelling and taking over the capital.
The sheer surface area that AI can create in our intellectual life is limitless
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
At some level of abstraction it's all built around allowing you to do some work/play/etc that you understand. Some of that work allows you to make money and eat food.
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
If you replace "LLM" with "mathematician" than this is the state of the world today. Stuff like Galois theory is beautiful mathematically, but what has it constructed or enabled for you and me?
There is a difference between manufacturing and design. Yes manufacturing is largely done by machines because of the nature involved but to look at a cpu and think 'no human was involved in creating this', you're wrong and also insulting to the humans who actually worked on things that led to the cpu and the manufacturing process.
A cpu (the physical thing that sits in your mother) is not an abstraction, what are you talking about
I think it is like saying what is the point of playing chess when you can never beat stockfish?
What is Magnus Carlsen going to do when he can't beat the computer?
It seems like a category error between humans using tools and humans building tools.
There is not much point in trying to figure out a better chess engine. There has never been a better time though to want to learn chess.
I find the idea that the computer will discover mathematics and humans call it a day rather ridiculous. As if humans will not then spend their time understanding and incorporating the ideas from the computer.
Alphafold is a better example. Alphafold is only bad if you spent your life trying to solve protein folding. But even if you did, that is the same person who is the most setup to reap the benefits of the unlock in the pragmatic application of protein folding.
We don't figure out how to get machines to harvest corn and then spend all day sitting around eating corn in between naps.
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Or maybe ants and humans are qualitatively different. Maybe there's a critical mass of intelligence where you can pretty much understand anything, and maybe humans are past that threshold. I don't know that for sure, but I don't think we're anywhere close to hitting fundamental limits to our ability to understand the universe.
I think a better analogy would be comparing to an ancient human instead of an ant. An ancient human would have none of the basic abstractions that we take for granted today like literacy and arithmetic, so it would be very difficult for them to even attempt trying to understand quantum mechanics. But I don't think it's impossible because our ability to learn by stacking abstractions is basically endless——so far as our health permits at least.
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
Ha, are you invoking something he said prior to discovering Russell's paradox? Which came about because he wanted to show that all of mathematics can be derived from logic about sets, but instead he showed that it couldn't be.
As someone with a math background, I don't see Russell's paradox as some interesting mathematics beyond set theory. It's just an example showing that one should be careful about defining sets.
As someone with a math background, and a logic background, I think Russell's paradox is most interesting, beyond just set theory. It shows us that one should be careful about what exactly a definition is.
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...
Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
One wonders whether a generation that demands instant satisfaction of all its needs and instant solution of the world's problems will produce anything of lasting value. Such a generation, even when equipped with the most modern technology, will be essentially primitive — it will stand in awe of nature, and submit to the tutelage of medicine men.
There's no reliable evidence that Neanderthals were less intelligent than modern humans. They're extinct now (except for a tiny genetic legacy in some human populations) but that could have happened for a variety of reasons unrelated to intelligence or lack thereof.
Calculators and computers are deterministic, they give the same output to the same output every time. Language models specifically are not. So it might give you are function that is correct, or a function that is not, or worse yet a function that behaves correctly, but introduces some god-awful bug down the line that can cause serious havoc. It is obvious that they need supervision, not only for output, but also sandboxing and various harnesses for them to not do any “oops, I deleted your codebase sry” kind of nonsense people post to Reddit.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.
So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.
You are confusing hardware noise and whether the algorithm itself is verifiable as if parity bits were not a thing. We don’t trust computers because the medium itself is infallible, but because the logic is sound.
With statistical models - such as LLM’s - there is no logic as such, but statistical assumptions based on given data. The output can ge very good or very bad, but you are fool to trust it blindly. Therefore you need a deterministic way to verify, whether meat- or software-based.
Why will X not keep increasing? It’s been increasing, albeit very slowly, since the start of the 20th century as the disease burden declines and nutrition improves. I see no reason to think that future health interventions couldn’t further juice those numbers.
Will it keep up with Y? Probably not, unless people are willing to accept pretty radical interventions to their biology. But it almost certainly is not static
If you set the temperature 0, an LLM is also deterministic (same prompt -> same output every single time). We just don't do this because the LLM is not so smart in that mode. But "LLM is not so smart" is changing at Moore's law speeds.
Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.
Ehm, no! The same prompt does not always lead to the same output.
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
You're missing the forest for the trees here. The point OP is trying to make is that calculators originally were essentially non-deterministic. Technology will go from unreliable to reliable.
LLMs are not non-deterministic by design. The randomness in the output is artificially injected for ergonomic reasons. (Yes, the non-determinism in production systems is different, but also not by design)
You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.
What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
The more I think about space elevator, the less I believe in it.
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
the economically dominant strategy to not verify them and not double check them
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
My dad recently needed to buy a new thermostat for his home with an air furnace (yes, he told the model) asking an AI which one to buy, and he got recommended one that only properly works with boilers. Then after that happened, the alternative he bought the AI never told them he needed to buy a gateway to connect to his furnace.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
Research in pure mathematics is part of what we call "basic research". There are no applications in mind a priori. People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere. It's the same as theoretical physics or theoretical computer science.
I think you have misunderstood the OP's point here. You're arguing that deepening human understanding is an end in itself, and you are right. The OP is arguing that advances don't need to be pegged to human understanding, and they are right too. The two can coexist, superintelligence far ahead of us, pioneering discoveries - and mathematicians catching up at a pace suited to biological minds. I don't see the issue here. Of course, it does mean mathematicians adopt a new role as hobbyists.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
For even more people fun is TikTok, Snap, Instagram -> sounds like a collapse of a civilization to me if you increase the ratio even more towards dancing kids sharing their content non-stop with no added value to the society
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.
Frontier AI is far beyond "hype-laden PR". You're right that there's a long way to go in release terms before fusion plants, but at the current tempo that 'long way' looks near in human terms. Whatever age you are, would you bet against it arriving in our lifetimes?
I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it
LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
What if programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks are already good enough, and the innovation lies elsewhere?
You can do a lot of cool stuff with the same lego pieces.
The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
I guess, the possibility of reclaiming human meaning and purpose.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
Indeed, and that's not what I said or was implying. My comment was an analogy in response to:
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny.
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
It's entirely possible in 2026, to further manually optimise compiler generated assembly, but vanishingly few people do that.
The point of my comment is that 'good enough' is almost here as demonstrated by the parent's comment.
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
Your “analogy” doesn’t hold up. The scrutiny applied to compilers are done by the compiler developers. Eventually if requirements don’t change the full test suite becomes the oracle. Not because of an attestation from a ghost in the machine but because of scrutiny done, let’s say over two years on a compiler that was reaching feature parity.
This obviously holds for compilers generating correct code since it is so well defined.
And this also holds for the efficiency of the generated code, since that is also obviously scrutinized by compiler developers.
Granted, the venerable LLM and the compiler do meet in a sort of functional intersection where all you can concievably care about is some thing that has a well-defined test for functionality or fitness. In the compiler’s case that’s the benchmark (good enough to not look at the assembly). But then one should go to that example directly and not to compilers in general.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue.
When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
but there is a difference between deterministic compilation and non-deterministic LLM code
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.
I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
I pored over every single line of code Claude generated with razor sharp scrutiny.
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.
If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
There is a failure to understand that the process is the result. You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind. The output of an LLM is useless without a human mind to comprehend it. We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact. Practice, applied over a lifetime, is what creates the capability for comprehension. Asking an LLM to give you an answer creates an artifact. Humans being humans, most of their requests boil down to "make me rich without having to work for it," so the request itself is paradoxical and impossible to satisfy. Philosophers have only been saying this for all of human history, so don't hold your breath for any breakthroughs.
nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.
The desire for novelty and to be first is demand. The publishing and grant writing. The whole funding and tenure apparatus. It's nothing but demand gradients.
People who study STEM for the enrichment of their intellect are privileged. Most of us had to study STEM (or whatever else) because we had bills to pay.
The output of an LLM is useless without a human mind to comprehend it.
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.
mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
mRNA vaccines were created by humans who understood them
If authors understood them, why most of their predictions were wrong? Time to decompose spike proteins, side effects, distribution through brrast milk....
Event the most basic promise 100% protection against infection was wrong! They lied and we allowed infected people into hospitals!
Precisely. I don’t really care whether LLM’s can produce code more and better than me. I learn languages, program and study computation to understand better the world we live in. Being a human being means increasingly being technologically augmented. To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.
History teaches us that understanding deep problems in mathematics finds natural applications elsewhere. So, either AI completely solves mathematics (unlikely for the moment), or we use marginal amount of funding to keep the mathematical communities alive.
To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.
We seen in last decades how "experts" are clueless, and how they predictions work.
don’t really care whether LLM’s can produce code more and better than me
So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?
You are both reading too hastily to quite grasp what I am claiming as well as making claims and assumptions you do not back with evidence. If there is tons of evidence how a person with LLM can understand - I was speaking of the world and times we live in, not just single problem - a problem an expert does, it is quite trivial for you to present that evidence.
Mind you,solving a problem is not same as understanding it. Anyone can just take a science calculator and bunch in some function without understanding anything about how to solve it.
The output of an LLM is useless without a human mind to comprehend it
This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.
You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.
Not exactly. The input to produce commodities is labour-power (the potential to perform labour). Education or study is the process of producing the commodity of labour-power, the potential labour of a mathematician who can do some sort of specific work.
It’s not ‘of course false’, and, to be honest, I can only assume you have little idea what mathematicians actually do.
Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).
Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.
I think this is the most beautiful letter on the subject I've read all year, it brought a tear to my eye. It's like reading those famous STEM letters/essays from history.
I agree. I believe it could be the best piece about AI ever written. The themes of lost dreams reawakened, sobering up of egos, sci fi technological development, the role of humanity. It’s a weird feeling to see that it’s actually really happening.
If we assume that trusting a model to execute an action is an ongoing exercise, since its trustworthiness is often discovered by the user organically as models develop, should trustworthiness be measured at the level of the model, or at the level of the human intent behind triggering it, whether explicitly or implicitly?
What the article really says is that we're going to need much smarter mathematicians.
That is not possible for puny meat-brain humans. Humans are close to their ceiling. AIs are just getting started.
In practice, we're probably going to hit that limit first in IC design. I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU, and it took about 5,000 engineers at peak to design it.
Getting that many people coordinated on one thing was a real achievement.
Then Intel stayed with minor tweaks on that design for years.
We're soon going to be seeing designs of even greater complexity cranked out by AIs. No human will understand them at the gate level. Reading AI-written programming language code is bad enough. Reading AI-written Verilog may be beyond human comprehension, except in small sections.
I think understanding the output of AI will become more and more likely impossible. It's not necessarily a bad thing, if we can benefit from AI produced knowledge, even if we didn't get there ourselves.
To take on this responsibility, we may need to broaden our view of what a mathematician can contribute. I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
Our ability to understand difficult and unfamiliar ideas may become one of the most important contributions we can offer to society. We should be willing to bring that skill to problems far beyond our usual research interests. [3] Doing so asks us to expand our sense of our vocation.
Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.
In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.
A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.
We need curious and ethical people.
Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.
One question: LLM might answer correctly many logical and math problems, but I don't see any guarantee that when the LLM context receives new information the answer could get worst. Security in LLM answer is not a monotone increasing function of context size.
I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
I can’t afford to be one of your credentialed reserves, Amit. ADHD screwed me over in early life or I’d have breezed through that dual masters in cognets/compsci by 2003, and now that I’ve stabilized my brain and am burning through years of school at a rapid pace, my country’s socioeconomically ruined — I’ll be lucky to escape with my accounting degree as my school is visibly being sucked into the vortex every year I progress. The only hope left for me to be what you need of us is to self-study, but without a degree I’ll just be treated as a crank or an AI proxy/puppet if I slip up and talk about interesting math with anyone, so what’s even the point of taking that path? I originally pivoted my math skills into systems theory and process diagnostics instead, which of course now everyone has kicked to the curb and replaced with AI. I’d have made an excellent Susan Calvin — I was working to be a cognitive tech with diagnostics, zeroth law, and group psych specialties, AMA! — but the financial investment to provide the runway to take that lonely, dreary six year slow as molasses slog through maths that universities think is somehow an appropriate teaching velocity (six months for precalculus alone?!) in order to earn the mere chance to have my resume rejected by an AI firm that uses AI hiring and and can’t tolerate someone with a strong moral position regarding societal harms is a very bad choice, whether you use simple probability or game or systems theory to evaluate it. Taking that quarter-mil-plus burden as loans in the hopes of employment at the other end in a field actively having its social, reputational, and moral fabric being ripped apart by AI? That’s not just a bad bet, that’s chasing fool’s gold at the end of a fading rainbow in a desert mirage. So, with sincere apologies, while you’d benefit from having me on your ‘reserves’ list (I have written testimonials spanning some twenty-plus years to that effect) I’ll never come to your attention as a support tech or as an accountant, and I accepted that outcome years before anyone else realized this need for cogsci mathematicians with a teacher’s specialty of analogy-building and the ability to disregard interpersonal nonsense to focus on the needs of societies. Better luck with the next generation, though!
"If AI fulfills its promise, we will encounter more beautiful and consequential ideas than we have ever seen."
It will be as it always has been: these ideas will further enrich the rich, at the expense of everyone else. We'll have our first quadrillionaire, while the masses are debating whether the minimum wage of $7.25/hr should be bumped up.
What a doomer way of thinking. Scientific advancements has positively improved the lifes of people in the past I don't see how scientific advancements accelerated by AI would be any different.
A lot of the conversations around AI in the past years have been “we will delegate almost all work to AI, but the most decisions will be left to humans.”
I’m fairly confident that the opposite will be true. The most important decisions will be made by AI, and humans will only be left to guide decisions as a matter of taste. AI has the ability to be impartial, and immutable. You can endlessly probe and reason its decisions.
This isn’t true of our current human decisions. Where we see red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
I do like the idea that we will need more mathematicians, physicists, and scientists to understand the discoveries of AI. That might be the best possible outcome of AI, but I fear the opposite and we are painting ourselves into a corner of which we no longer have the knowledge to sustain ourselves and society itself collapses.
TT is politely and gently trying to deal with the ego of many mathematicians, namely to teach them humility, which they will need to design their specialized maths AI models(agents?). Well, dunno if it will work out anything interesting, but he is giving a try.
I wonder if the current maths models, if any, are able to use formal solvers in their 'reasoning'. I wonder if a natural language interface is really that efficient, maybe a pure formal language hinted with intuitive "tokens". I remember the time I was learning real maths: "elegant", "brutal", "strong", etc were somewhat meaningfull.
Is that already economic delamination? As in the singularity extracts the most valuable ressources (new thought generators) from the traditional economy keeping mankind alive and stable- towards itself?
Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
Certainly humans can enjoy exploring such things, but will they really be needed?
What does needed even mean? To produce shareholder value? Maybe not. To explore thoughts and ideas humans have for fun? Certainly
To make money in order to eat food and have a home.
Money will become unnecessary. The future Anthro Park will no doubt provide a safe enclosure, feeding at regular intervals, and an enriching environment for you and your fellow semi-evolved simians.
The current approach to resource distribution worked so far. When it stops working because we have too much abundance, we'll settle on another one.
That doesn't mean the transition will be smooth. It doesn't mean that certain classes won't be worse off. I can certainly imagine programmers being among the losers. And there certainly are futures where we slide in dystopia. So let's try building the good futures. Alea iacta est.
If AI leads to an era of abundance, then the economic system has to change. The system it transitions to and what that transition looks like will need to be fought for, and I imagine it’ll be done better in some places than others. Yet I think the parallel existential question of meaning in this world is one that is inherently optimistic. Humans decide meaning, it is not something imposed on us or taken away by the existence of something else.
Right now that looks like a bunch of rich edgelords and us as serfs
I don't think "abundance" is a stable state. There are always ways to use more resources and more energy, so no matter how much we have, it will never be "enough" -- abundance is just a transition state where a population or system has not yet figured out how to grind the surplus. The faster the progress, the shorter I would expect these periods to be.
How do you think this will work? Do you think the whole of the rest of the world, about 7 billion people, will agree to give up on wanting to reach the western life standard that is currently consuming 4 Earth's of resources every year?
Will all rich people in all countries collectively agree to be moved into tiny apts same size and give up their luxuries?
If not, how do you define who gets what? Who gets to live in the fancy house, etc?
This is beyond naive.
We could feed the whole population of the world if we decided to, we definitely have enough production[1]. If we pretend AI-assisted automation can do most work - which still currently is an highly dubious claim - we could just provide food and shelter for free to everyone, if we so desired. It would require quite a lot of restructuring and dietary changes, but it is possible.
[1]https://online.ucpress.edu/elementa/article/doi/10.1525/elem...
If this is true, you would think that companies would be opening shops in the middle of (some shithole country) so they can make money (or someone else would grab it). But they don’t, why is that?
Because the shithole countries are poor and rife with violence that is hostile to operating a profitable company? Obviously?
Referring to countries as "shithole countries" does not help in understanding the current dynamic between rich and poor nations. So perhaps it is helpful to understand what that is, without the pejoratives.
Firstly, each country is different. So the relationship between say the USA and Mexico is different to the relationship between the USA and say Ethiopia.
By "richness" we might also be better off thinking as less "spare cash" and more "standard of living". Which includes food, housing etc but also security, safety etc.
In truth much of the US "richness" depends on other nations. If there was no Latin labor the US would have a fresh produce problem. Indeed the very premise of DJT is that the US has a trade deficit with just about everyone. Much of US "richness" is thus dependent on the "poorness" of others.
Yes automation, cheaper energy, AI can reduce the labor content of goods and make them universally cheaper. Yes I believe in a future where the overall standard of living increases.
But it would be a mistake to believe the US is "leading" US there. The US achieved its current position at the barrel of a gun, and uses the military (and threat of military) to interfere with countries and regions either overtly or covertly. The entire premise of the cold war (outside of western europe) was premised on interventions to resist countries switching their political model to something the locals preferred.
And yes, maybe the locals preferred wrong. But the US didn't promote democracy, they promoted "willingness to keep supplying us". Countries which were leaning away from the US had to be punished. (Shah in Iran, Vietnam etc.)
I could go on, but I'll stop there. Suffice to say that an equitable standard of living is not in the (current) interest of most US citizens.
Shithole country? You mean US: https://www.msfoodnet.org/about-us/hunger/
The companies do not do this, because equal distribution of resources is poison to capitalism. The world where everyone are fed, and their basic needs - shelter, medication etc. - are taken care of, does not have trillionaires. Possibly not even billionaires.
It is not profitable to give people humane standards of living. You work a lot harder when you see the homeless on the street on your way to work.
You're right that we could already feed everyone with a different distribution of produced wealth. That part is not something that AI will change (although it would raise the resulting standard of living).
What it does change is the basic equation underlying capitalism as an economic system: a few people concentrate capital in their hands, but capital by itself is useless without labor, so they have to hire workers to produce something useful with all that capital that they own. The workers get a shit sandwich, comparatively speaking (given the amount of wealth they generate), but they do get something out of it. The system is stable overall because, while a better arrangement is possible, most people - even low-wage workers - have too much to lose to riot.
With AI that can do most work, the capital no longer needs labor. People who own all the robots can just have them produce whatever they need. Everybody else is then "economically redundant", and the whole system collapses because people aren't going to sit around and starve because there are no longer jobs to be had - and at this point they have nothing left to lose and everything to gain by rebelling and taking over the capital.
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
And this will be true for how long? 3-4 months?
As long as there are problems to discover and solve, which is forever.
The risk of liability is a social problem that is far more difficult to be solved with technical solutions even with AI.
Even if that happens, what’s the point if there’s no human involvement? AI doing math for math’s sake? And doing what with it?
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
At some level of abstraction it's all built around allowing you to do some work/play/etc that you understand. Some of that work allows you to make money and eat food.
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
If you replace "LLM" with "mathematician" than this is the state of the world today. Stuff like Galois theory is beautiful mathematically, but what has it constructed or enabled for you and me?
There is a difference between manufacturing and design. Yes manufacturing is largely done by machines because of the nature involved but to look at a cpu and think 'no human was involved in creating this', you're wrong and also insulting to the humans who actually worked on things that led to the cpu and the manufacturing process.
A cpu (the physical thing that sits in your mother) is not an abstraction, what are you talking about
Yes, why not? And, of course, AI doing math for AI.
We may not be needed forever...
Stuff! Inscrutable stuff, maybe, but that's not "doing math for math's sake."
I think it is like saying what is the point of playing chess when you can never beat stockfish?
What is Magnus Carlsen going to do when he can't beat the computer?
It seems like a category error between humans using tools and humans building tools.
There is not much point in trying to figure out a better chess engine. There has never been a better time though to want to learn chess.
I find the idea that the computer will discover mathematics and humans call it a day rather ridiculous. As if humans will not then spend their time understanding and incorporating the ideas from the computer.
Alphafold is a better example. Alphafold is only bad if you spent your life trying to solve protein folding. But even if you did, that is the same person who is the most setup to reap the benefits of the unlock in the pragmatic application of protein folding.
We don't figure out how to get machines to harvest corn and then spend all day sitting around eating corn in between naps.
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Don't write us out quite yet. :)
Or maybe ants and humans are qualitatively different. Maybe there's a critical mass of intelligence where you can pretty much understand anything, and maybe humans are past that threshold. I don't know that for sure, but I don't think we're anywhere close to hitting fundamental limits to our ability to understand the universe.
I think a better analogy would be comparing to an ancient human instead of an ant. An ancient human would have none of the basic abstractions that we take for granted today like literacy and arithmetic, so it would be very difficult for them to even attempt trying to understand quantum mechanics. But I don't think it's impossible because our ability to learn by stacking abstractions is basically endless——so far as our health permits at least.
This is an unfortunate example to choose, being as it is entirely confounded by the canonical illustration of the https://en.wikipedia.org/wiki/Law_of_triviality.
How is that relevant at all?
How is it not relevant? The construction directly contradicts what Sahai imagines happening, and the worked example is the nuclear power plant.
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
Ha, are you invoking something he said prior to discovering Russell's paradox? Which came about because he wanted to show that all of mathematics can be derived from logic about sets, but instead he showed that it couldn't be.
As someone with a math background, I don't see Russell's paradox as some interesting mathematics beyond set theory. It's just an example showing that one should be careful about defining sets.
As someone with a math background, and a logic background, I think Russell's paradox is most interesting, beyond just set theory. It shows us that one should be careful about what exactly a definition is.
Life is all about tradeoffs. Mathematics have no tradeoffs. We need more engineers.
Life is all about purpose. Mathematicians are instead concerned with meaning. We need more philosophers.
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Bugs have existed since before AI, though. It's remarkable how bad a lot of very successful software has always been.
Sure, but we currently find that acceptable. Codebases are full of such bugs.
Nit, but Terrence Tao did not write this article, it’s a guest post.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
- Eric Hoffer
- Tech Bro
There's no reliable evidence that Neanderthals were less intelligent than modern humans. They're extinct now (except for a tiny genetic legacy in some human populations) but that could have happened for a variety of reasons unrelated to intelligence or lack thereof.
were gonna need a citation on this one.
Calculators hallucinate! Mine did not come with error correcting RAM. (Though you might not know from the price.)
Calculators and computers are deterministic, they give the same output to the same output every time. Language models specifically are not. So it might give you are function that is correct, or a function that is not, or worse yet a function that behaves correctly, but introduces some god-awful bug down the line that can cause serious havoc. It is obvious that they need supervision, not only for output, but also sandboxing and various harnesses for them to not do any “oops, I deleted your codebase sry” kind of nonsense people post to Reddit.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
How do you know?
Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.
So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.
You are confusing hardware noise and whether the algorithm itself is verifiable as if parity bits were not a thing. We don’t trust computers because the medium itself is infallible, but because the logic is sound.
With statistical models - such as LLM’s - there is no logic as such, but statistical assumptions based on given data. The output can ge very good or very bad, but you are fool to trust it blindly. Therefore you need a deterministic way to verify, whether meat- or software-based.
Humans (IQ of X, non deterministic) can write deterministic code.
AI (IQ of Y, non deterministic) can write deterministic code.
Y is going to keep increasing, while X will not.
Why will X not keep increasing? It’s been increasing, albeit very slowly, since the start of the 20th century as the disease burden declines and nutrition improves. I see no reason to think that future health interventions couldn’t further juice those numbers.
Will it keep up with Y? Probably not, unless people are willing to accept pretty radical interventions to their biology. But it almost certainly is not static
IQ hasn't been increasing in developed countries for about three decades now. In fact, it has been declining in most.
The increases still happen globally but mostly driven by developing countries.
If you set the temperature 0, an LLM is also deterministic (same prompt -> same output every single time). We just don't do this because the LLM is not so smart in that mode. But "LLM is not so smart" is changing at Moore's law speeds.
Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.
Ehm, no! The same prompt does not always lead to the same output.
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
LLM output is very much not deterministic!
These are all implementation details.
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
Nothing is deterministic.
Nothing is stopping LLMs to be more deterministic/correct over time.
Also you yourself is nondeterministic :)
All people are. That is how automation appeared to begin with - to provide deterministic behavior.
You're missing the forest for the trees here. The point OP is trying to make is that calculators originally were essentially non-deterministic. Technology will go from unreliable to reliable.
Calculators never were non-deterministic by design.
LLMs are not non-deterministic by design. The randomness in the output is artificially injected for ergonomic reasons. (Yes, the non-determinism in production systems is different, but also not by design)
It will be even more amazing if they solve the theory of everything or the hard problem of consciousness.
Imagine AI crushing quantum mechanics like Einstein pwned classical physics.
You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
This assumes we're still building them on Earth.
Why would we have stopped?
No, it doesn’t. It assumes we’re using it here. If we beam the power to Earth from space, it’s the same thing.
We use it in space too, we just send food and manufactured products down the elevator.
You aren't really trying in good faith to think this through are you? This idea is over half a century old. Not getting it by now is willful.
The more I think about space elevator, the less I believe in it.
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
I feel like knowing something is true is useful, but if you don’t understand how and why, you won’t understand the implications
The models get things wrong in the way that humans don't.
They will never make a logical error yet make terrible assumptions and poor long scale decisions.
Wake me up when an agent swarm can write gcc in a box sealed from the internet.
My dad recently needed to buy a new thermostat for his home with an air furnace (yes, he told the model) asking an AI which one to buy, and he got recommended one that only properly works with boilers. Then after that happened, the alternative he bought the AI never told them he needed to buy a gateway to connect to his furnace.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
it's incredible the amount of people who think all these recent posts in Taos blog were written by him.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
Another day, another HN thread full of programmers who think mathematics is just like programming.
Thanks for providing a (much needed!) correction.
Well with AI, programming among humans will become like math. A hobby.
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
Research in pure mathematics is part of what we call "basic research". There are no applications in mind a priori. People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere. It's the same as theoretical physics or theoretical computer science.
I think you have misunderstood the OP's point here. You're arguing that deepening human understanding is an end in itself, and you are right. The OP is arguing that advances don't need to be pegged to human understanding, and they are right too. The two can coexist, superintelligence far ahead of us, pioneering discoveries - and mathematicians catching up at a pace suited to biological minds. I don't see the issue here. Of course, it does mean mathematicians adopt a new role as hobbyists.
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
But suppose some future holy grail AI can do much more than that.
Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.
But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
Isn't it fun to imagine how life would look like in that scenario?
We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
You can't just leave us hanging like that!
Published in 1909:
https://www.gutenberg.org/cache/epub/72890/pg72890-images.ht...
https://en.wikipedia.org/wiki/The_Machine_Stops
Successfully neutralized by AI, I suppose.
no
I think the weirdest part of that scenario is that science might become something closer to archaeology
We would care about having fun.
For some people, fun is doing physics and mathematics. So they are going to keep doing that.
For even more people fun is TikTok, Snap, Instagram -> sounds like a collapse of a civilization to me if you increase the ratio even more towards dancing kids sharing their content non-stop with no added value to the society
This is how most people already live.
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
In the case of medicine it goes deeper than that, sometimes nobody really understands why a medication works.
STEM will be considered "historical studies" in an AI-ruled future.
Look at the financially desolate subcultures with no option for advancement or dignified life.
That is the goal and that is how it will lool like, if the tech CEO managed to gain the power they want.
Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.
Frontier AI is far beyond "hype-laden PR". You're right that there's a long way to go in release terms before fusion plants, but at the current tempo that 'long way' looks near in human terms. Whatever age you are, would you bet against it arriving in our lifetimes?
I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it
LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
What if programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks are already good enough, and the innovation lies elsewhere?
You can do a lot of cool stuff with the same lego pieces.
That is like saying what if music is already good enough.
This is a guest post by Amit Sahai.
The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
It has to be said a million times. A fully deterministc compiler (or 99.99% or whatever) is categorically different from an LLM.
Hopefully this million plus one mention shifts the right weights around the datacenters.
Indeed, and that's not what I said or was implying. My comment was an analogy in response to:
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
It's entirely possible in 2026, to further manually optimise compiler generated assembly, but vanishingly few people do that.
The point of my comment is that 'good enough' is almost here as demonstrated by the parent's comment.
Well, there's the rub. Humans and LLMs that asked to solve a problem at a higher level will rarely write the same code twice. Write the simplest regex, and you won't come up with this https://www.cs.princeton.edu/courses/archive/spr09/cos333/be...
The future is indeterminism.
Your “analogy” doesn’t hold up. The scrutiny applied to compilers are done by the compiler developers. Eventually if requirements don’t change the full test suite becomes the oracle. Not because of an attestation from a ghost in the machine but because of scrutiny done, let’s say over two years on a compiler that was reaching feature parity.
This obviously holds for compilers generating correct code since it is so well defined.
And this also holds for the efficiency of the generated code, since that is also obviously scrutinized by compiler developers.
Granted, the venerable LLM and the compiler do meet in a sort of functional intersection where all you can concievably care about is some thing that has a well-defined test for functionality or fitness. In the compiler’s case that’s the benchmark (good enough to not look at the assembly). But then one should go to that example directly and not to compilers in general.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue. When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one. I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
There is a failure to understand that the process is the result. You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind. The output of an LLM is useless without a human mind to comprehend it. We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact. Practice, applied over a lifetime, is what creates the capability for comprehension. Asking an LLM to give you an answer creates an artifact. Humans being humans, most of their requests boil down to "make me rich without having to work for it," so the request itself is paradoxical and impossible to satisfy. Philosophers have only been saying this for all of human history, so don't hold your breath for any breakthroughs.
That's nonsense. Of course you study them to meet demand.
nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.
The desire for novelty and to be first is demand. The publishing and grant writing. The whole funding and tenure apparatus. It's nothing but demand gradients.
sad
People who study STEM for the enrichment of their intellect are privileged. Most of us had to study STEM (or whatever else) because we had bills to pay.
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.
mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
If authors understood them, why most of their predictions were wrong? Time to decompose spike proteins, side effects, distribution through brrast milk....
Event the most basic promise 100% protection against infection was wrong! They lied and we allowed infected people into hospitals!
Precisely. I don’t really care whether LLM’s can produce code more and better than me. I learn languages, program and study computation to understand better the world we live in. Being a human being means increasingly being technologically augmented. To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.
Great but you will no longer be paid to do what you love.
History teaches us that understanding deep problems in mathematics finds natural applications elsewhere. So, either AI completely solves mathematics (unlikely for the moment), or we use marginal amount of funding to keep the mathematical communities alive.
I have never been paid to do what I love. Well, I got couple of art grants some years ago.
There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.
We seen in last decades how "experts" are clueless, and how they predictions work.
So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?
You are both reading too hastily to quite grasp what I am claiming as well as making claims and assumptions you do not back with evidence. If there is tons of evidence how a person with LLM can understand - I was speaking of the world and times we live in, not just single problem - a problem an expert does, it is quite trivial for you to present that evidence.
Mind you,solving a problem is not same as understanding it. Anyone can just take a science calculator and bunch in some function without understanding anything about how to solve it.
This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.
Your analogy doesn't work. A "product" in research pure math is not the same as writing code.
Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.
Not exactly. The input to produce commodities is labour-power (the potential to perform labour). Education or study is the process of producing the commodity of labour-power, the potential labour of a mathematician who can do some sort of specific work.
It’s not ‘of course false’, and, to be honest, I can only assume you have little idea what mathematicians actually do.
Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).
Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.
I think this is the most beautiful letter on the subject I've read all year, it brought a tear to my eye. It's like reading those famous STEM letters/essays from history.
I agree. I believe it could be the best piece about AI ever written. The themes of lost dreams reawakened, sobering up of egos, sci fi technological development, the role of humanity. It’s a weird feeling to see that it’s actually really happening.
If we assume that trusting a model to execute an action is an ongoing exercise, since its trustworthiness is often discovered by the user organically as models develop, should trustworthiness be measured at the level of the model, or at the level of the human intent behind triggering it, whether explicitly or implicitly?
"We’re gonna need a lot more mathematicians."
What the article really says is that we're going to need much smarter mathematicians. That is not possible for puny meat-brain humans. Humans are close to their ceiling. AIs are just getting started.
In practice, we're probably going to hit that limit first in IC design. I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU, and it took about 5,000 engineers at peak to design it. Getting that many people coordinated on one thing was a real achievement. Then Intel stayed with minor tweaks on that design for years.
We're soon going to be seeing designs of even greater complexity cranked out by AIs. No human will understand them at the gate level. Reading AI-written programming language code is bad enough. Reading AI-written Verilog may be beyond human comprehension, except in small sections.
If Math Academy teach 10 year old kids calculus, I doubt that.
I think understanding the output of AI will become more and more likely impossible. It's not necessarily a bad thing, if we can benefit from AI produced knowledge, even if we didn't get there ourselves.
The point of the post is that what you call "AI knowledge" is not knowledge.
Not Terrence Tao post. Beware.
Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.
In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.
A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.
We need curious and ethical people.
Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.
One question: LLM might answer correctly many logical and math problems, but I don't see any guarantee that when the LLM context receives new information the answer could get worst. Security in LLM answer is not a monotone increasing function of context size.
you arent a real mathematician unless you lay around all day thinking about shapes
I can’t afford to be one of your credentialed reserves, Amit. ADHD screwed me over in early life or I’d have breezed through that dual masters in cognets/compsci by 2003, and now that I’ve stabilized my brain and am burning through years of school at a rapid pace, my country’s socioeconomically ruined — I’ll be lucky to escape with my accounting degree as my school is visibly being sucked into the vortex every year I progress. The only hope left for me to be what you need of us is to self-study, but without a degree I’ll just be treated as a crank or an AI proxy/puppet if I slip up and talk about interesting math with anyone, so what’s even the point of taking that path? I originally pivoted my math skills into systems theory and process diagnostics instead, which of course now everyone has kicked to the curb and replaced with AI. I’d have made an excellent Susan Calvin — I was working to be a cognitive tech with diagnostics, zeroth law, and group psych specialties, AMA! — but the financial investment to provide the runway to take that lonely, dreary six year slow as molasses slog through maths that universities think is somehow an appropriate teaching velocity (six months for precalculus alone?!) in order to earn the mere chance to have my resume rejected by an AI firm that uses AI hiring and and can’t tolerate someone with a strong moral position regarding societal harms is a very bad choice, whether you use simple probability or game or systems theory to evaluate it. Taking that quarter-mil-plus burden as loans in the hopes of employment at the other end in a field actively having its social, reputational, and moral fabric being ripped apart by AI? That’s not just a bad bet, that’s chasing fool’s gold at the end of a fading rainbow in a desert mirage. So, with sincere apologies, while you’d benefit from having me on your ‘reserves’ list (I have written testimonials spanning some twenty-plus years to that effect) I’ll never come to your attention as a support tech or as an accountant, and I accepted that outcome years before anyone else realized this need for cogsci mathematicians with a teacher’s specialty of analogy-building and the ability to disregard interpersonal nonsense to focus on the needs of societies. Better luck with the next generation, though!
It will be as it always has been: these ideas will further enrich the rich, at the expense of everyone else. We'll have our first quadrillionaire, while the masses are debating whether the minimum wage of $7.25/hr should be bumped up.
What a doomer way of thinking. Scientific advancements has positively improved the lifes of people in the past I don't see how scientific advancements accelerated by AI would be any different.
Humility is the wrong word. We didn't feel it when the steam engine was introduced, so why now?
It really looks like Tao is on the path of accepting and embracing AI now.
Maybe he is, but you are quoting Amit Sahai, the author of this blog post.
Are you willing to pay for a lot more mathematicians?
Or it as with other professionals: shortage means shortage of cheap labor.
A lot of the conversations around AI in the past years have been “we will delegate almost all work to AI, but the most decisions will be left to humans.”
I’m fairly confident that the opposite will be true. The most important decisions will be made by AI, and humans will only be left to guide decisions as a matter of taste. AI has the ability to be impartial, and immutable. You can endlessly probe and reason its decisions.
This isn’t true of our current human decisions. Where we see red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
I do like the idea that we will need more mathematicians, physicists, and scientists to understand the discoveries of AI. That might be the best possible outcome of AI, but I fear the opposite and we are painting ourselves into a corner of which we no longer have the knowledge to sustain ourselves and society itself collapses.
Which AI has this ability? LLMs definitly don't.
I recommend you the book the rational optimist.
Not saying you’re wrong you just can’t be so certain
TT is politely and gently trying to deal with the ego of many mathematicians, namely to teach them humility, which they will need to design their specialized maths AI models(agents?). Well, dunno if it will work out anything interesting, but he is giving a try.
I wonder if the current maths models, if any, are able to use formal solvers in their 'reasoning'. I wonder if a natural language interface is really that efficient, maybe a pure formal language hinted with intuitive "tokens". I remember the time I was learning real maths: "elegant", "brutal", "strong", etc were somewhat meaningfull.
Is that already economic delamination? As in the singularity extracts the most valuable ressources (new thought generators) from the traditional economy keeping mankind alive and stable- towards itself?
I imagine:
* Massive budget overruns
* Year and year of delays
* It ending up wasting more energy than it produces, or just not working, period
* Politicians disclaiming responsibility and blaming it on "the AI"
* The contractor companies profiting immensely over the entire period, and ending up not having to worry about maintenance, warantees, etc.