Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
With Fable 5, Strands harness cost 77% less than Claude Code and scored higher on Terminal Bench 2.1.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling.
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
Can you also compare in the charts https://maki.sh?
Should give you some competition.
why would you include oh-my-pi in the comparison but not vanilla pi?
Am I wrong in saying that the interfaces presented to the model in OMP versus plain old Pi are identical?
OMP has a lot of candy that raises token cost compared to vanilla pi
Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
Why is Pi not in the benchmarks? Deepseek beats Strands and its built on Pi so that’s all I needed to know.
Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
The sales pressure from them on their agent core stuff has been really shocking over the last six months. Never seen anything like it.
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
Yes. Or- use them at work, where management is taking a... hands off approach to ~integrating ai~ into the workplace.
For what it's worth I've been using Anthropic models on Pi for months now with no issues. It's not recommended since it breaks TOS but you can do it.
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
I’m sure it was not a coincidence that this was released the day after Kimi 3 was added to bedrock.
Will they block me if I build an agent with this that shops on Amazon?
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
[delayed]
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.