- 48comments
- 26comments
- 6comments
- 399comments
- 109comments
- 64comments
- 240comments
- 341comments
- 16comments
- 269comments
- 266comments
- 54comments
- 323comments
- 27comments
- 8comments
- 1comments
- 80comments
- 3comments
- 2comments
- 29comments
- 66comments
- 2comments
- 20comments
- 63comments
- 38comments
- 59comments
- 40comments
- 78comments
- 482comments
- 198comments
"When evaluating the Intelligence Index, it generated 140M tokens, which is somewhat verbose in comparison to the median of 140M."
Nowadays these error can be a good thing :)
Human error means this wasn't just stopped together by some bot.
My bet is that it's a bot error, but of a rule based one.
Yes, it seems they have a template that they fill with numbers. Similar issues spotted on Grok's performance page: https://news.ycombinator.com/item?id=49789558
It feels suspicious that MiMo-V2.6 Pro gets 46 in de index while DeepSeek-V4.1 (https://artificialanalysis.ai/models/deepseek-v4-1-flash) gets 39. According to the appendix at the bottom of https://mimo.xiaomi.com/mimo-v2-6 the deepseek model sometimes surpasses mimo and it's not so far behind in capabilities. A week ago opus 5 appeared 1 points ahead of fable 5 despite fable being a much smarter model (this has been corrected already)
Why?
The main AA benchmark keeps changing, and had to be radically changed when Astra came out and showed zero improvement over GPT 5.6 Sol in their benchmark. Opus 5 is still 1 point ahead of Fable 5.0 on the index, if you manually add Fable 5.0 back into the list, so it hasn't actually been "corrected". It's only Fable 5.1 that is shown as ahead of Opus 5.
The AA benchmark is a weighted average of other benchmarks and some internal ones. I think the difficult part is finding benchmarks that reflect your own use of the models.
It is an impressive model. Agreed on most that is written on this page, with the exception of it being fast. I ran it on my own LLM benchmark suite[1] and it is faster than DeepSeek but still much slower than leading models. But it's pricing is where it really shines.
KillSwitch-Bench 1.0
1 - https://bench.killswitch-lang.org/
Why sol is not in the comparison?
the graph has a dropdown for selecting models
OpenAI usage limits have been severely cut, and intelligence appears to be markedly declining, so I'm going to start trying these Chinese models seriously now. I don't mind if it takes longer. I just need the intelligence to predictably work the same way from day to day.
Same- I pay $200/mo for Codex but whereas I used to get a week's work out of a weekly limit, now I get roughly 1~2 days.
I've stopped using Astra entirely and remain on Sol orchestrating Luna Xhigh, but it's still not nearly a week's usage for a week's allotment.
And even then, whenever a new model is about to come out, it feels like the model I'm using is being dumbed down substantially.
I have no evidence for this and can have no evidence for this, but I can vote with my wallet regardless.
I strongly agree. Check out the Codex subreddit. Many empirical examples of Astra silently downgrading the models. One found Astra was silently using Luna Max (but still billing for Astra).
Even when I try to stick with Sol X/High, my limits are at best half of what they were before Astra launched, and the intelligence has declined markedly.
I cancelled my $100 plan. This is absolutely absurd and frankly unusable now.
Per Xiaomi, MiMo v2.6 training run cost $3.47m. A far cry from the estimated costs ($100m+) for the Big 5 (MSL, xAI, GDM, OAI, Ant). I wouldn't be surprised if salaries and R&D costs have similar drastic disparities.
For a model that matches Muse Spark 1.3 in benchmarks, MiMo v2.6 Pro is incredibly cheap, given its cache rates will remain $0.0036 per million.
I sorta got the impression that the $3.47 million only covered post-training , given that few of the graphs start at zero. Is a barely-trained model going to score 48 on DeepSWE v1.1 ?
https://mimo.xiaomi.com/rl/