Article: "How Good Are Frontier Models at Physics?
Expert Re-Grading Reveals Broken Evaluations and
Near-Saturation of Leading Benchmarks"
John Sous from Yale posted a fairly solid study on how nearly all the physics benchmarks are broken, as they consistently mark correct answers as incorrect.
When hand grading instead, they found out that the models have actually already saturated the benchmarks which is a little bit scary.
Article: "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks"
John Sous from Yale posted a fairly solid study on how nearly all the physics benchmarks are broken, as they consistently mark correct answers as incorrect.
When hand grading instead, they found out that the models have actually already saturated the benchmarks which is a little bit scary.
This is interesting and actually very important for robotics.
I've been waiting for this, but all companies seem to not care much now.
There is a way out of this by supplying right context (needs a bit of expertise in physics)
1 more year and frontier will become crazy good at this as well.