I actually explored this for robotics last year, and CLM grew directly out of that work a few months ago. Feel free to check out my earlier post for more details on how CLM could be applied to robotics:
https://x.com/jackyk02/status/2026368947210289660
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
I wonder when work started on this project, and how the public release of Jev played into their timing.
From a tweet:
https://x.com/jackyk02/status/2102947585208258799?s=20
But of course the repo with the TypeSafe compatible API is new: https://github.com/Contrastive-LM/CLM
Does the latency even matter?
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
I tried TypeSafe’s Jev playground. It outputs the model latency and network latency separately. The model latency was 100-200ms in my tests.
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.
I really hope that "System One" won't stick around as a new buzzword simply meaning "fast".
It already did, since 15 years ago.
https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
i'm well aware of the origin and meaning of the term
this implies more than just "fast", which is precisely why i don't like its present usage
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong?
CLIP[1] for actions! Very cool.
[1] https://openai.com/index/clip/
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.