Because vanilla pi doesn't do a lot of important things out of the box, and they didn't want to open the "which extensions should we add" can of worms.
Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
I'm using Opencode Go in OMP or Hermes. $10 a month and I have only ever hit a limit using qwen3.8MAX on X-High. This is a migration from 20x on Claude.
I am using Pi with anthropic models for plan & execution, openai models for code review and fallback in case I run out of tokens. Also have custom hooks to automatically compact, switch models based on the type of tasks, change effort and custom statusline. Also have custom setup in Pi to calls tools and CLIs instead of relying on mcps. The level of customization I can reach with pi is not really feasiable with codex or claudeCode. By the way anthropic changed it's ToS regarding custom harness: you can`t use their model unless you got extra usage enabled or paying through APi billing, you cant use your subscription anymore. We all guessed what the reason might be. Also codex is open source, I havent really looked at that project yet because Pi fulfill my desires for now.
With Fable 5, Strands harness cost 77% less than Claude Code and scored higher on Terminal Bench 2.1.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling.
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
Yes, it’s a wild claim. I built a harness for a random side project without even thinking hard about it. The harness was that from the hardest part of the project.
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
Beyond that though, I'm certainly interested in the performance side of things. "Keep an eye out for a follow-up paper from our researchers regarding these benchmarks." Yes plz.
I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
This matters less as models get better and everyone settles on the same overall harness architectures. The model matters more than the harness anyway.
The bigger issue is that the use cases and harnesses for models is infinite, which is hard to compress into benchmark numbers that actually apply to you.
Everyone is benchmaxxing, desperate to sell, and almost nobody except the labs is doing actual science on the results, so harnesses tend to be chosen on voodoo and hunches, like which company made it. There isn't necessarily a good alternative though, bearing the cost of being a harness researcher is probably not many people's goal.
Agreed, especially since the more frontier models are able to accomplish in a vacuum, the more people will trust them. That being said, tool use is still really important for pulling in the right information.
I remember reading about strands SDK and it looked great in terms how everything is an event that you can extend, so this harness feels quite about right.
However, for this kind of customisation, Pi is actually quite great. One of the most things I love about Pi is ability to ask it to create an extension and it does it quite well as it’s part of their docs. Also ability to customise the system prompt to avoid the clutter that Claude Code add (around 20k system prompt that mostly had nothing to do with the code).
The demo was showing something I have created for my Pi setup, which is asking me in each new session which skills and MCP I would to enable for the session. This works quite well if you have multiple projects where you don’t need all skills but just a small subset
finally paying up for chatGPT and using the Codex desktop app was my real "I'm sold" moment with AI.
Setting up projects and working with the AI on local files has been great, but only for my personal account. I've been trying to get it set up for work that provides OpenAI models through a 3rd party tool, company hosted models, as local-machine models and the UX is just straight up awful.
there's no GUI for profiles or custom endpoints, the config.toml sucks and the overall experience is primitive.
At this stage I really want a Codex-like harness but I need more fluid control over the models, I want features like pinning a project to a provider, as well as pulling in all the models from that provider, also having all providers available.
So if I need to pop over to one project to consult about product A, then pop to another project to do some code analysis on product B I can do so fluidly and have my tokens billed to the right place for each concern.
or a project that can span all of the resources. like having the OpenAi models orchestrate sub-agents on the local or hosted models.
so far I have seen many harnesses that are built to provide similar quality to the provider's harness for lower cost. Has there been any projects that prioritizes quality and quality consistency?
Can you also compare in the charts https://maki.sh?
Should give you some competition.
why would you include oh-my-pi in the comparison but not vanilla pi?
Am I wrong in saying that the interfaces presented to the model in OMP versus plain old Pi are identical?
OMP has a lot of candy that raises token cost compared to vanilla pi
Because vanilla pi doesn't do a lot of important things out of the box, and they didn't want to open the "which extensions should we add" can of worms.
Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
Why is Pi not in the benchmarks? Deepseek beats Strands and its built on Pi so that’s all I needed to know.
Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.
That’s fair, but Strands is advertising their harness needing way less tokens which does relate to cost.
In that same vein, Pi is less bloated then Deepseek and Oh My Pi, which are built on top of Pi. Isn’t it dubious to leave it out?
Deepseek harness is not actually built on Pi harness. It's an independent project.
This is true, but the team said on Twitter that they were heavily inspired by Pi and this whole team used it extensively:
https://x.com/tianyi/status/2088306143772946499
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
The sales pressure from them on their agent core stuff has been really shocking over the last six months. Never seen anything like it.
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
Yes. Or- use them at work, where management is taking a... hands off approach to ~integrating ai~ into the workplace.
For what it's worth I've been using Anthropic models on Pi for months now with no issues. It's not recommended since it breaks TOS but you can do it.
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
I'm using Opencode Go in OMP or Hermes. $10 a month and I have only ever hit a limit using qwen3.8MAX on X-High. This is a migration from 20x on Claude.
That’s exactly what I do. OpenAI + Opencode Go subs, 0 interest in Claude.
OpenAI. It avoids Claudish too.
pi on open weights, I only use frontier models to do a review pass
I am using Pi with anthropic models for plan & execution, openai models for code review and fallback in case I run out of tokens. Also have custom hooks to automatically compact, switch models based on the type of tasks, change effort and custom statusline. Also have custom setup in Pi to calls tools and CLIs instead of relying on mcps. The level of customization I can reach with pi is not really feasiable with codex or claudeCode. By the way anthropic changed it's ToS regarding custom harness: you can`t use their model unless you got extra usage enabled or paying through APi billing, you cant use your subscription anymore. We all guessed what the reason might be. Also codex is open source, I havent really looked at that project yet because Pi fulfill my desires for now.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
I’m sure it was not a coincidence that this was released the day after Kimi 3 was added to bedrock.
Will they block me if I build an agent with this that shops on Amazon?
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
Yes, it’s a wild claim. I built a harness for a random side project without even thinking hard about it. The harness was that from the hardest part of the project.
Is this corporate confabulation?
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
Beyond that though, I'm certainly interested in the performance side of things. "Keep an eye out for a follow-up paper from our researchers regarding these benchmarks." Yes plz.
I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
This matters less as models get better and everyone settles on the same overall harness architectures. The model matters more than the harness anyway.
The bigger issue is that the use cases and harnesses for models is infinite, which is hard to compress into benchmark numbers that actually apply to you.
Everyone is benchmaxxing, desperate to sell, and almost nobody except the labs is doing actual science on the results, so harnesses tend to be chosen on voodoo and hunches, like which company made it. There isn't necessarily a good alternative though, bearing the cost of being a harness researcher is probably not many people's goal.
Agreed, especially since the more frontier models are able to accomplish in a vacuum, the more people will trust them. That being said, tool use is still really important for pulling in the right information.
I’m the opposite. I want one open source harness to rule them all
Cost efficiency is a plus
This seems to be from AWS team. Is that right?
I remember reading about strands SDK and it looked great in terms how everything is an event that you can extend, so this harness feels quite about right.
However, for this kind of customisation, Pi is actually quite great. One of the most things I love about Pi is ability to ask it to create an extension and it does it quite well as it’s part of their docs. Also ability to customise the system prompt to avoid the clutter that Claude Code add (around 20k system prompt that mostly had nothing to do with the code).
The demo was showing something I have created for my Pi setup, which is asking me in each new session which skills and MCP I would to enable for the session. This works quite well if you have multiple projects where you don’t need all skills but just a small subset
finally paying up for chatGPT and using the Codex desktop app was my real "I'm sold" moment with AI.
Setting up projects and working with the AI on local files has been great, but only for my personal account. I've been trying to get it set up for work that provides OpenAI models through a 3rd party tool, company hosted models, as local-machine models and the UX is just straight up awful.
there's no GUI for profiles or custom endpoints, the config.toml sucks and the overall experience is primitive.
At this stage I really want a Codex-like harness but I need more fluid control over the models, I want features like pinning a project to a provider, as well as pulling in all the models from that provider, also having all providers available.
So if I need to pop over to one project to consult about product A, then pop to another project to do some code analysis on product B I can do so fluidly and have my tokens billed to the right place for each concern.
or a project that can span all of the resources. like having the OpenAi models orchestrate sub-agents on the local or hosted models.
I their web icon a pepe frog? hard pass if so..
so far I have seen many harnesses that are built to provide similar quality to the provider's harness for lower cost. Has there been any projects that prioritizes quality and quality consistency?