The link rgbrgb posted is a good overview. The best open ones are close to Jev now, but they're big models. And I agree, if you have an eval set for a fixed task, a trained classifier is the better choice.
It is not. In a benchmark with actual decisions - navigation, traffic, waypoints - laya does slightly better than a small classifier, with very low correlation to state changes.
I am fairly confident if Jev-style decision models are seen as prominent (which, they seem to be), Ollama will support them. Surprised the team hasn't implemented this already.
Depends on the model. The small ones I support today are well below Jev on harder queries, but fine for simple, well-defined questions. The open models that get close to Jev are bigger, and I'm adding support for those next.
Has anyone actually seen better or the same results with Laya compared to Jev? From my experience, Laya performs significantly worse. It's less confident and often makes wrong decisions with more complex queries.
Developer here. You're right, Laya is a lot weaker than Jev, especially on harder queries. It's a small model, so it's fast, but that's the trade-off. The open models that get close to Jev are much bigger, and running those is what I'm working on next.
Yes. JEV generalizes better because they probably have an enormous corpus and trained on it for a long time. Laya's out of the box model is much weaker. However, in the age of LLM's it's incredibly easy and cheap to generate large datasets to fine tune laya for your task, and the training loop is pretty quick and cheap too.
It's so easy that I question why I would ever pay for JEV when eventually I'll have done enough random things that I will also have a large corpus and likely a general model as well.
Nothing yet. Unfortunately it sometimes feels like our industry has been overrun by grifters and chancers.
I’m sure this has been a gradual and long decline. Maybe it even started with the dot com boom and accelerated with crypto. With AI it seems to have got worse.
Are there many models that are comparable to Jev for generic decision making?
Smarter move if you have an eval set is to just train a classifier and call it a day.
there's this thing with a bunch of similar models https://huggingface.co/spaces/multimodalart/jev-decision-ind...
top open one is trained by perplexity cto for $3k, kinda cool https://x.com/denisyarats/status/2102252088067850507
<<<"i was curious to see if i could train a competitive Jev-like model completely autonomously with a swarm of agents using our internal system."
Bro is writing off the H200 lol
On a sidenote I really can't stand the term "swarm" and definately plays into AI doomerism.
The link rgbrgb posted is a good overview. The best open ones are close to Jev now, but they're big models. And I agree, if you have an eval set for a fixed task, a trained classifier is the better choice.
Cool... but this does seem undermined by the fact that Ollama can add support for decision models at any time.
Fair, and I'd be happy if they did. Ollaya uses the same API as Jev, so your code isn't tied to it either way
and that ollama is go-llama and not rust, so it's not really the ollama of anything
great project for empowering open-source alternatives.
open-source is the only way for safe AI development. whoever doesn’t share the weights/code will lag behind.
Thanks!
Fair point, that example is basically classification. I'll change it to something that looks more like a real decision.
text classification is equivalente to decision. This is exactly the same thing Jev does.
Jev does it more efficiently because it doesn't use an LLM https://typesafe.ai/blog/introducing-system-one-models-and-j...
It is not. In a benchmark with actual decisions - navigation, traffic, waypoints - laya does slightly better than a small classifier, with very low correlation to state changes.
If it has four legs, a tail and barks why not call it a dog?
"Decision model" is just marketing jargon.
decision model = classifier
system one model = small non-reasoning LLM
noul = boolean
confidence = f(probabilities)
It's sad to see how gullible engineers are today.
I am fairly confident if Jev-style decision models are seen as prominent (which, they seem to be), Ollama will support them. Surprised the team hasn't implemented this already.
Sounds good on latency but how is its actual decision quality vs. Jev?
Depends on the model. The small ones I support today are well below Jev on harder queries, but fine for simple, well-defined questions. The open models that get close to Jev are bigger, and I'm adding support for those next.
Has anyone actually seen better or the same results with Laya compared to Jev? From my experience, Laya performs significantly worse. It's less confident and often makes wrong decisions with more complex queries.
Developer here. You're right, Laya is a lot weaker than Jev, especially on harder queries. It's a small model, so it's fast, but that's the trade-off. The open models that get close to Jev are much bigger, and running those is what I'm working on next.
Yes. JEV generalizes better because they probably have an enormous corpus and trained on it for a long time. Laya's out of the box model is much weaker. However, in the age of LLM's it's incredibly easy and cheap to generate large datasets to fine tune laya for your task, and the training loop is pretty quick and cheap too.
It's so easy that I question why I would ever pay for JEV when eventually I'll have done enough random things that I will also have a large corpus and likely a general model as well.
Nothing yet. Unfortunately it sometimes feels like our industry has been overrun by grifters and chancers.
I’m sure this has been a gradual and long decline. Maybe it even started with the dot com boom and accelerated with crypto. With AI it seems to have got worse.