Haven't gone through full PDF as its very detailed, few things have resonated with me so far.

Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.

You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).

Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.

It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.

"Everybody knows foul air causes sickness."

"Duh, of course Mars has canals."

Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.

using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses

not really, there was a recent benchmark with claude and codex and it showed no difference in ability with a harness like pi agent compared to their native harnesses, pi was in fact cheaper per task.

bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash?

No. The conclusion is that:

bash-capable models + bash-only tools > bash-capable models + predefined tools

In other words, MCP was just a bunch of bullshit that maybe helped a little bit until the models got good at bash, and now it's basically useless.

Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back"

Anthropic appears to agree frontier models don't need in-session planning tools.

https://github.com/anthropics/claude-code/issues/80487

I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.

I'm going to cherry pick one example where newer models are noticeably improving at least in my experience.

What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.

I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup