I'm sure there's a simple answer to this, and I'm probably just missing it, but what happened with the supergravity one?

And how many runs did it take before this one? They say "in one shot" with nothing more than "keep going." But we only see the successful run, reported by the people who ran it.

I feel like usage of AI alone should be enough to uncap it, so long as AI companies do the predictable thing and steal everything from everyone.

There are thousands of people figuring out how to make AI better for their particular thing. Way more than that walking the AI through taking things from problem to solution.

AI can totally identify optimizations within a paradigm without human intervention. What it isn't good at now is deciding to try totally new paradigms. Humans aren't great at this either, though it's more cultural in our case.

I'm an AI skeptic, or even anti-AI, and even I would have a hard time to admit it hasn't gotten any better.

In fact I will even admit it, yes AI has gotten better... but also how could it NOT? It has literally all the resources it can has, namely attention from everyone, everybody talks about it, a lot of workers, from state of the art researchers to annotators, all the material resources, from dedicates chips to networking to water to electricity.

AI literally has everything humanity can provide, it better be "better".

AI can now do really hard science homework all by itself without messing it up, which is kinda cool.

Cool, that was an interesting read on its own. I viewed the html version and skimmed it, I appreciated your illustrations.

What if the shore isn't a clean line?

The training costs get rolled into the usage prices. It's not hiding anything to talk about the cost of usage without the cost of training, any more than I'd be hiding something by talking about the price of a $1,000 CPU without mentioning the tens of billions of dollars in R&D and infrastructure needed to create it.

Also love: scientists get to pay Anthropic to work within their special science harness to do science. That's exactly what I dreamed of doing when I pursued physics in undergrad, one or two companies holding the keys to "progress" for a monthly subscription price.

I get this anxiety, and am largely an AI skeptic, but at one point there were only a handful of computers in the world too (same for batteries, or engines, or crucibles, or stills -- it goes way back), and the organizations that had them had a stranglehold on progress in the field, as did the small number of companies who knew how to make them. It got better as they got cheaper and more plentiful.

I guess my point is that there are more important anxieties to feed when it comes to LLMs and the current state of the world.

This feels like an obvious jab at Navier-Stokes, but where we get to shift the numbers around to hide where the compute actually is being spent ... compute is being spent. It's either being spent in amortization to make the search smarter ahead of time, during training, or its being spent after.

I think that argument is recursive? These posts aren't very complicated for either of us, but they're written on devices that are fabricated with billions of dollars of semiconductor equipment. At what point do we just acknowledge that we stand on the shoulders of giants?

To me, the distinguishing factor is that the expense not special-purpose but upfront. The model here is trained without foreknowledge of what problems it will solve. Solutions like nine loops are genuine expressions of a pre-existing model capability, even if that capability has not pre-existed for very long.

What a weird thing to be butthurt about. You don't have to pay them. Do it the old fashioned way, with elbow grease and pots of coffee. Or invest in local AI.

Or embrace the future and realize that you couldn't imagine everything that would unfold, when you pursued your undergrad.

How you gonna get mad about all this?

Divide the compute of your argument (pre-training, training, post-training) through everything this model now can do and it will not look that bad at all.

normally i'd agree, for "most people"

but the author explicitly distinguished between "AI" and "LLM-based AI", and they work for an AI company, where making these distinctions are really important

Fair point, and i think that's fine. AI labs could just not jump themselves into such problems and let humans have a go at them, whether ai-assisted or not. As in this case with a regular budget one researcher could've access to.

I think it's mostly just what it is on it's face; You're telling it to be independent for a while. The default prompts start in an interactive mode where it checks in and verifies requirements a lot.

Smart model in charge, planning & orchestrating the dumb models doing the work.

Often dumb models from other plans & subscriptions. I like to choose whatever model is on sale on opencode-go.

YMMV drastically based on what the actual work is. Dumb models can't handle everything. And of course I have no idea how much of a backlog of work you're feeding it or anything. Some people have enough work to exceed any plan, and do the work inefficiently to boot.

Also, you can just buy as much usage as you want at API rates.

First of all, there is definitely value addition with the LLMs in almost every field in some ways.

What bothers me the the marketing angle which invites skepticism and criticism

Anthropic invited Matt von Hippel to write this post and compensated him for his time.

If you are paying some one, tailoring the discussions then the end result is always going to be biased one showing yourself as the winner. I understand its somewhat organic, still the ratio of marketing and science needs to be balanced. Marketing has to be correct and the results/outcomes should be reproducible

only scratching the surface. The impact of AI might be significant/belligerent/virulent

I don’t understand what the model actually did here or how it was verified, and I feel like that’s a general problem with all these “AI did breakthrough science” posts. I understand that I’m not an expert in these fields so there’s only so much I can pick up from a blog post about it, but these posts are basically always just “trust us, it did the thing.” I mean surely you could easily share the conversation trace and any artifacts/code it produced so peers in the field can verify the work.

These scattering amplitude formulas are hard to compute, so hard that physicists almost always use approximations. They do partial calculations, cut off at a specific number of “loops,” a measure of how complicated interactions between particles are allowed to get. The more “loops” they include in their calculations, the closer they get to the real answer, and the harder, computationally, the calculation is to do.

I don’t even understand what type of solution we’re describing here, is it a formula? A program? A Lean proof?

It may have gotten a boost from using Python, and not Maple (Lance’s favorite program for math) or Mathematica (mine), and it may have used much better software engineering practices than we would have, but not super-intelligently so.

This is the crux of the matter here. Physicists are physicists. They are not software engineers. I read physics between 2001 and 2005, and the programming language they had us use for all of our assignments was FORTRAN 77. They were still trying to decide if it was ok to move students on to FORTRAN 95.

FORTRAN is pretty performant - if you know what you’re doing. Very, very few people did - and they, me, went on to have careers in software, not physics. I remember some FITS (astronomical image format) processing software someone was using to calculate ephemera - and it took DAYS to run over a few thousand images and produce an output. I sat down with it, screwed around for an afternoon, and it produced a result in under a minute - so much faster that I honestly thought I had broken it - but I hadn’t. It was just terribly written by someone who was excellent in their domain and terrible at writing code.

I see there being an enormous opportunity here, in AI providing scientists with software that isn’t diabolical.

[delayed]

I want to see AI beat a 4x strategy video game or a roguelike. Let me see a 20+ win streak on the hardest difficulty in Balatro or Slay the Spire. Let me see AI beat Civilization against the best human players.

Every time I mention this someone assures me that it's possible, and they point to simple board games that computers excel at, or real time strategy games where proper use of APM and clicking accurately go a long way. But I haven't seen AI succeed at any decision-focused game where describing the rules requires more than one minute.

I want to see what AI can do in, not simple, and not complex, but complicated toy environments, where decisions are all that matter.