i really like this. the “the agent finished” vs. “the agent actually did what it was supposed to do” distinction is something i don't see addressed very well by most observability tools.
traces, tokens, latency etc. are great, but in the end i care about whether the job got done. i think this is a pretty interesting way of looking at it. - and it is free, right?
It is totally free and open-source we are looking for the first design partners to ship the enterprise version on production. The enterprise repo is using the OSS as the core and make it easily scalable. There are some differences in the technologies used tho.
The way I do it is actually by Defining Yaml contracts, to define the business outcome so using the traces you can understand if it reaches the goal. Evaluations also play a big role
i really like this. the “the agent finished” vs. “the agent actually did what it was supposed to do” distinction is something i don't see addressed very well by most observability tools.
traces, tokens, latency etc. are great, but in the end i care about whether the job got done. i think this is a pretty interesting way of looking at it. - and it is free, right?
It is totally free and open-source we are looking for the first design partners to ship the enterprise version on production. The enterprise repo is using the OSS as the core and make it easily scalable. There are some differences in the technologies used tho.
Cool tool, now I know how much I've spend on AI xD. But how exactly can you measure if the outcome of a flow was successful?
it's probably magic
The way I do it is actually by Defining Yaml contracts, to define the business outcome so using the traces you can understand if it reaches the goal. Evaluations also play a big role