archive link in case i get hugged lol https://archive.ph/Z4gxF

I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.

They do…? Well, “auto” mode has been default in Claude Code for a couple months now

They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.

That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.

I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.

This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.

Liability. They want you to be the one who enabled it when it deletes your production database.

What even is superintelligence? Is my phone not a superintelligence?

Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.

the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse

when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns

There are probably fewer fields where you can verify rewards than one might hope.

2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.

"Any reason why that can't be solved through context management and keep-forward scaffolding?"

becomes

"load bearing context seam"

/s

Remember specifics, keep going.

I think automation is coming but it will be way more gnarly than frontier labs want public to believe. Value is just too big, when you can automate most of eg customer support it will create huge savings and same time customer satisfaction will get better.

My belief is that LLMs will fundamentally change how we approach domain expertise. From what I've seen, SDEs tend to be over-specialized compared to what the company actually needs to implement due to the need to understand enough of the domain to pick a best path. If an LLM can see the domain enough so that someone in an adjacent field can be confident in their approach and quickly change course then you don't need as many niche SDEs

We've had technology beating humans on memory for millennia, and we've had technology beating humans on computation for many decades now.

The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.

Also doesn't the very good ARC AGI 2 score of GPT-6 Astra kinda contradict this, since each problem is its own game with very different rules

Yes, we all know that biological systems are more efficient than machines through billions of years of evolution and natural selection but the overall process is largely the same (interacting with an environment, learning from results, improving underlying architecture, etc); efficiencies will come with more time and improvements.

but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?

Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.

Isn’t this exactly how AlphaZero was trained? The rules are known and well defined so the training process can generate games without any outside data.

The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.

The caveat is: It depends on the task.

I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.

The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.

Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.

Are there reams of chess moves that the model can train off of? No.

For real??

There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.

I wonder how current models would fare. The ones they tested are fairly old now.

more like you lose intelligence in chess by maxing for coding... hence knocking back the claims of emergent intelligence

I feel like every position is in the database so you could just lookup the most popular move for an arbitrary elo and that's the bot.

Probably tells us that without labs explicitly training/tuning the models or designing the harness (with fast oracle) the LLMs aren't going to get good at those areas.

As someone who used to compete for years and plays currently as a hobbyist, you’re absolutely correct. LLM’s are terrible at chess and if anyone wants to sober up their view on AI, try it yourself.

Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.

Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous

So you can see an actual game on that website, and the play seems pretty decent to me for a while (~1700 lichess = 1300 elo) until move 28 when black throws away their queen for absolutely no reason in an incomprehensible blunder.

In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.

It's not even a "software-generating product". It's only half of it. Most of the heavy lifting is done by absolutely not-AI compilers, analyzers and the like. If not for these programs, written well before AI boom, them LLMs would be no better at programming than they are are at pure LLM based calculations or writing.

An AGI doesn't stand for 'perfect intelligence' it stands for artificial general intelligence.

And no an AGI system doesn't need to play chess on a certain level to be disruptive to you and me and whole industries. It only needs to be as good as a person and cheaper.

Just because you define AGI as something it doesn't has to be,doesn't mean i need to touch grass.

This chess comparision is one of the most ignorant and stupid arguments i have heard after the parrot thing

I believe I remember reading it was based on Glaurung's code (which eventually evolved into what we now know as the juggernaut Stockfish).

I'm pretty sure "competent at chess without external aids" has been on the standard AGI checklist since before personal computers were a thing. How can you claim an intelligence is general if it can't make sense of such a highly constrained board game? This is solidly table stakes.

First, you're moving the goalposts. Second, it's not actually true that any existing frontier AI can write a chess bot program that can beat a 1600 player ... not unless the program is derived from Stockfish or some other leading engine that has been in development for decades.

The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.

These comments indicate a complete failure to understand the technology.

I won't respond again.

State-sponsored psyop meta comments aside, the models obviously continue to get better, but there is still a lot of 'guard railing' required to keep even the latest models completely on-task. The chess example is interesting because it's clearly a well-studied and established domain so the rules, strategies, and whatever else is in the training data should make yield excellent results; but clearly there is some behavior in these systems that's difficult to engineer out.

Exactly, especially when you touch “forbidden things”, like questioning why rust IS not the best system programming language, you will be punished so hard by “expert”s.

Back in 2001, our social medium was Slashdot and no one ever pretended to read the article. No one read the article either. It was slashdotted most of the time anyways.

even if I give them literal infinite time and all the subagents and internet access..

Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.

HackerNews is Gell-Mann amnesia that refreshes on every comment on every thread.

I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access.

I don't believe this.

You refer to "subagents", so this is not just an LLM but an LLM with some kind of agentic harness. Any reasonable harness and prompt, given internet access and appropriately prompted to succeed on this task, is more than capable of firing up Lichess or chess.com and relaying moves back to you. The free levels will be enough to beat you.

A frontier model can also likely one shot a chess engine that plays at your level, again if given an environment in which it can do that.

I completely believe the LLM on its own can't play a full game of chess at your level. Though I'd bet that with enough reinforcement learning it is possible to train a pure transformer architecture to do that. We just don't do it because there are other approaches that play chess much better.

"Elo is relative to the ChessBench field."

They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.

If it's a GM then I'm Magnus Carlsen, https://lichess.org/study/27lCQqDa.

Dumbest thing I’ve seen today

Please do not post misinformation. They are not playing anywhere near GM level.

"Elo is relative to the ChessBench field."

I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.

They still need supervision though

Well I guess this excuse is finally gonna become obsolete soon with all the "pacing" nonsense.

Is it? I am on the fence on this, but it does seem like there are some qualitative improvements between the models.

Not related to your post, but a fact I keep mulling over. The fact I don't trust the current crop of LLM's enough and I consider LLM's as a tech will hit a ceiling pretty hard, it doesn't mean parallel improvement curves won't spring up out of other research that will lead to much higher capabilities than currently.

No, but superhuman capabilities derived from ordinary learning is, which is what the parent comment described. Why is that not obvious?

And how many years of direct play and study does it take for a human to get good at chess or any other game?

Time is irrelevant to training; the more relevant comparison is "how many games does a human need to play to get diminishing returns".

A week. My brother learned and was above 1100 online within 12 hours, after a few hundred games.

Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess.

Maybe not, but you'd be surprised how little it takes.

A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.

This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.

Poorly in what sense? I think human chess leagues are way more popular and fun than just playing a computer by yourself. Human oriented communities are always a vastly better experience than their digital counterparts.

There's more to games than simply winning you know.

Comparing to raw LLMs? Much much better.

If they wanted to train an LLM to play chess they could easily do so.

But nobody wants that.

Asking an LLM to play chess by writing algebraic notation is like asking a human to play chess blindfolded.

Yes some people can do it but most people can't even if they're unusually intelligent.

You really need to be giving the LLM a board representation.

EDIT: I see that they actually were giving the LLMs a board representation and they still played badly. Fair enough then.

That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.

The _only_ illegal move a human might make as a beginner is a failed en passant or a bad castle. And yes, a few hundred games is all it takes to be better than any publicly available LLM at the moment.

WTF even is this post?

Where is all the training and practice LLMs did to become so good at coding?

Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.

It's called post-training, typically through some form of reinforcement learning, and is a significant part of modern LLM development.

You have the first stage, pre-training, which is learning from next token prediction. That's where the model memorises a lot of facts about things and generally gets good at forms of writing. It's like reading a lot of books on programming and reading through a lot of source code. It's learning how to autocomplete code, essentially. Doing that requires a developing a reasonable understanding of code, but it's also learning how to autocomplete bad code as well as good, and won't make it a "good" programmer.

Pre-training uses a method called Cross-Entropy Loss to update the weights of the network.

Then comes post-training. This is where the model is trained against huge sets of example problems, like fixing a bug, adding a new feature based on a spec, etc. They are set the task and try to complete it inside a training environment. Once they're done, their complete solution is evaluated (either by humans, or by some separate evaluation model that was developed based on human feedback) and they are updated based on whether the solution was good or not.

Post-training uses a different method called Proximal policy optimization to update the weights of the network.

So these really are very different forms of learning, and mainstream LLMs are not post-trained to be good at chess. They could be. You could easily create a reinforcement learning environment that evaluated and improved their ability to play and win at chess. The result would be a very strong chess playing AI, something we know is possible because the strongest chess playing programs we have are neural network based, but it is not a priority for AI companies.

Not one mention. Hundreds of books, articles and databases of games.

OpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.

Humans don't need a lot of training and finite tuning to make only legal moves.

An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.

An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).

1100 is literally below the ELO you get by default as a beginner.

1100 lichess/chess.com does not represent real elo. I'm around 1400 online, I would still be unranked in the real world. The fact that I easily beat any model publicly available is not a great look for AGI.

Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.

Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.

That's a polite way to put it. :-)

Declarative knowledge is not the same as procedural knowledge. You can read as many chess tutorials, strategy documentation and game archives as you like, they won't make you good at chess until you actually start practicing chess.

The issue with the models isn't that they play a bad game, but that they persist in making illegal moves. An average intelligent human can be told the rules of chess and then play chess, badly, within the rules.

That would mean we should consider any human with coding knowledge a chess grandmaster, which is obviously not the case.

Good science, properly digested and presented takes time.

The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues

I don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.

Exactly. All these nerds saying cars make bad submarines. Well duh.

Because then it’s not playing chess, stockfish is?

The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.

"current frontier models need laborious oversight and guardrails on even the simplest tasks"

I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.

What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?

Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?

Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.

It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.

The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.

I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.

You are right. I am imprecise.

Languages allow a certain flexibility in their grammars - you can read a sentence without that adhering it exactly to the grammar.

Games and programming languages (including lean) does not allow this flexibility.

A very intelligent person would likely also reason in terms of probably outcomes before correcting a statement to adhering entirely to the grammar.

Certainly it must be like that, otherwise reviews in math was rendered moot.

Do we blame research mathematicians for not adhering to the grammar?

Why?

You might never have tried to program before, so I don't blame it on you.

But most programmers, even experienced ones, see grammar and type errors regularly.

How much support do we as humans need to get rules right?

I'm an expert in my field, read my comments, my gramma is shit.

The fact that they can play chess at all despite having no specific training for it blows my mind, and the fact it doesn’t do the same for many others shows just how far they’ve come and how fast.

I suspect (in a probably ignorant fashion) that this is because learning process has been reading a lot of algebraic chess notation (such as "1. e4 e5 2. Nf3 f6 3. Nxf6 gxf6 4. Qh5! +-") then, to play, generating more of it without considering the rules of the game. This is exactly how it's always felt to me when playing chess against LLMs. Sure, "1. e4 e5 2. Nf3 Nc3" looks innocent to somebody simply learning the syntax of algebraic notation, but that Nc3 by black is an illegal move.

An LLM is the wrong approach for playing chess.

If you give the same task to an exceptionally intelligent human, who does not play chess and has only heard about it in passing, then they would be beaten by every child who has looked at the rules for more than 10 minutes.

What kind of intelligence is "playing <____> but we don't tell you the rules" supposed to test?

Well, yes, PGN files have structure... But still, playing Chess with an LLM is so weird that I impulsively question the sanity of people attempting to do so. Do some people really believe training on TWIC PGNs would make an LLM a good chess player?

I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.

You said it like labs like open AI doesn’t know and don’t constantly make moves to prevent that undercutting

I haven't heard of the term "chaining zero-days". Now as a SRE I wont sleep.

As compute prices fall it gets easier and easier to make "frontier" models. So it's inevitable that commodity, open source models of equivalent capacity to today's "frontier" models will be available to the public. Remember this is just weights, anyone can download them and run it whenever they like. The only constraint is compute.

depends on the blast radius of zero-days, it's not like there's a continuous immediate release process for these models; they can eval internally before releasing publicly

Isn't that assuming that fix won't be implemented?

Zero days are valuable because they can be exploited but if the pace of exploitation is faster (which I'm not sure is the case), then the response WILL be faster, even if it means going offline. Institutions that won't will simply go offline by losing their data or becoming unprofitable due to ransomware.

Now for components that are core to the infrastructure, say OpenSSL, there is already a TON of attention and efforts, including red teaming, so it's not as if it's opening floodgates.

Sure low hanging fruits will get picked either faster or a at a larger scale, say a random outdated IoT device at your local flower shop, but for the rest, I don't think it's realistic to expect no response.

Security, digital or not, has always been an arm race. New threats means new responses specifically by incorporating the threat.

good post/take.

doomers gonna doom

refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting

I am bullish on AI. At some point well see some true advancements.

I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.

Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.

DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.

Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.

Summarise means different things to different people.

Thats a browser level task.

great, autonomous LLMs will fail. that's actually perfect. they work amazingly well when we're telling them what to do. no autonomy needed, no destruction of humanity. that's all win

You’re right; given that most of the money in the AI market is injected through OpenAI and Anthropic (which collect it through both selling equity and through customer revenue), the 7-8T is just a derivative of that.

I think you are committing the lump of labor fallacy [1]. Lots of jobs will disappear, but others will appear. Lots of things (both intellectual and material) that are produced nowadays by humans will be produced in the near future by AI. But humans will be needed to do new things.

Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.

[1] https://en.wikipedia.org/wiki/Lump_of_labour_fallacy

Oh for sure, this was a simplistic analysis assuming AI adoption caps out at some X% of job responsibilities where X << 100%.

Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.

As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)

You are comparing company valuations to annualized revenue (as approximated by some fraction of total knowledge worker compensation). Valuations are (roughly) based on the sum of all discounted future cash flows, not just the current year’s revenue.

It does not have to be 16% of all jobs, but 16% of any given job, i.e. AI stays in an augmentative role rather than a complete job automation. The simplistic analysis is if a tool makes you X% faster, that can be worth X% of your salary to your employer.

Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616

It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.

Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.

The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.

I don't think that AI companies can charge the same. The human workforce can charge these costs, because of scarcity. But AI systems won't be scarce, it's just a matter of who can run inference cheapest. Plus you still have the human workforce, which might be forced to offer their time for less money.

There's a leverage issue.

In one case (financial services) it's thought that expertise is valuable at V=S^2/b4 where V is value, S is skill and b capacity (the leverage available to the manager/expert. b erodes as it becomes harder to find examples of things that are not done well, so if you manage $1bn you might find lots of miss allocations that you can exploit with just that $1bn really effectively, but if you manage $10bn it's much harder to find good places for the extra $9bn. A low hanging fruit effect.

Anyway, that double hit - raw skill and the amount of times you can supply the skill makes the value of skill (V) convex, and it means that in a perfect market (heh heh heh) someone running $100bn is worth 1000's or maybe 10,000's of an average joe expert.

Now, if AI is trusted to run the top 0.1% of everything and has the skill to do it at human top level expertise, then your calc holds. If it's the case that it isn't then more than half of that value disappears. If it's not even top 1% then chop out another 25%.

That implies that we need a lot of trust and a lot of AI capability before these valuations stack up, and it also implies that all other competitors and incumbants are going away. I do not think that Citidal or Bridgewater are going to let Anthropic or OAI take them without a fight. They might lose - but there is a decent bet that they don't. I don't think that many professions like Lawyers or Doctors are just going to roll over and cede their monopoly rights to OAI or Anthropic either.

And you know compare Eliza with what an LLM can do today?

Or do i miss the point you are trying to do?

No. Silicon is on another level. Which is why the EDA verification is an industry on its own.

Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.

LLMs are basically multi-dimensional magic mirrors.

Depending on where you point them, they can be incredibly useful.

They can even be useful when you point them at each other (though increasingly difficult to get good results).

I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.

Depends on what customer support means.

Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.

came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.

So, if you are concerned about what the model does? Yeah.

The SOTA models are heavily 'guardrailed' today; they won't let you do all sorts of things.

if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?

This logic doesn't follow at all.

If their argument is that there is 10% chance of extinction then they also believe there is a 90% chance it won't.

I looked at the math and I think it's true. Remember revenue is just sales, not profit. These labs are shooting for > $1T valuations, which traditionally means your PROFIT is at least 1/20th or 1/30th of that (so let's say minimum 30B$/year PROFIT).

These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?

So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).

when you put these models in a loop, you are able to do more and more insane and cool things...

Please share some of these insane things that you speak of..

Won’t this change though?

the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.

- i have bearish from day 1

- i have no idea how anyone thinks the mighty next token predictor is going to eradicate diseases and eliminate poverty https://blog.florianherrengt.com/how-llms-work.html

- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"

a topic with an unusual level of specification

Solving cancer also has an unusual level of specification. Many real world problems have that characteristic.

The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.

That's the thing, it very much does NOT show us that. What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.

Which is rather precisely the point that the article is making!

If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer

As the saying goes, if my grandmother had wheels she would have been a truck.

Wasn't Navier Stokes solved by ripping off a researcher's private chat log?

Yea I get the bearishness from my own personal experience.

Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.

But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.

Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.

Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.

So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.

I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).

[1] https://en.wikipedia.org/wiki/Wicked_problem

~Gosh, getting downvoted for trying to help a stranger, what is wrong with you people?~ thanks for the upvotes, guess was just a misclick.

Yep, I had the same thought. As simple heuristic, text written in all lowercase is often just hot takes and so not worth taking the time to read.

LC;DR :P

" the lack of sentence capitalization makes it unnecessarily difficult to read."

If it had proper caps etc, people here would accuse it of written using LLMs.

You just can't win...

A visual mockup?

Don't you think novel scientific work recombines existing knowledge? Isn't the saying precisely of standing on the shoulders of giants?

That being said I didn't compare both, not sure why you brought that up. I specifically discussed about prototyping, quoting a specific sentence, not scientific research.

AI is just good at what it's got the most elaborate training data on. And by "good" I mean, is statistically most likely to spit something out that makes some kind of sense.

I don't know how well it is studied, but I suspect it is possible there is language-related complexity constraints to the effectiveness of the LLM algorithms. Like perhaps context-free grammar related problems with adequate training data can be more and more effectively solved, but maybe natural language related problems will not so much be effectively solved.

Would be curious if there is active research here.

Geez, why do you upper-case "LLM"?

[delayed]