I actually explored this for robotics last year, and CLM grew directly out of that work a few months ago. Feel free to check out my earlier post for more details on how CLM could be applied to robotics:
https://x.com/jackyk02/status/2026368947210289660
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
I wonder when work started on this project, and how the public release of Jev played into their timing.
From a tweet:
https://x.com/jackyk02/status/2102947585208258799?s=20
But of course the repo with the TypeSafe compatible API is new: https://github.com/Contrastive-LM/CLM
Does the latency even matter?
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?