It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of one general model doing everything.
The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.
Ah, the old "good, fast, or cheap; pick two" proves true once again.
Give it a few months.
Some of us want fast food
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?
It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
The point is the speed.
If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
Also kimi 2.6 at 1000tps (as of may), though when we reached out they had a >12 month waitlist and minimum 7-8 figure annual token spend.
[0] https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
yeah. K2.6 can run on insane speeds. So sad that they don't have K3 yet.
But it can apparently also run 5.6 Sol
Yeah but at “call to discuss pricing” rates
Or better: Qwen 2.8 27b
Unfortunately, the lack of an input cache discount makes it prohibitively expensive for most use cases that aren't one-shot prompts.
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
Which is wild because it does, in fact, do caching
I honestly think the diffusion LLM approach is a dead end
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of one general model doing everything.
it's stupid fast!
It's stupid and it's fast.
this feels like "we got the same benches as gpt-oss-120b but are also potentially slower while saying it is great"