- 111comments
- 2comments
- 41comments
- 156comments
- 43comments
- 225comments
- 140comments
- 17comments
- —discuss
- 16comments
- 308comments
- 132comments
- 36comments
- 166comments
- 282comments
- 33comments
- 73comments
- 52comments
- 146comments
- —discuss
- 40comments
- 259comments
- 63comments
- —discuss
- 14comments
- 8comments
- 193comments
- 467comments
- 316comments
- 547comments
Edit: Comment deleted - no longer useful.
Apparently they updated it perhaps based on your comment?
Oh - I see that now. It's possible I missed it originally - I did read the article but I was skimming quickly. Mea culpa, if so.
You can have 100 runs for the price of the Claude Max plan?
Or the price of a GPT Pro 20X plan... er, wait, except you can't go sign up for one of those right now.
I find this - or perhaps the title - a bit surprising.
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.
DS has perf issues if you parallelize it heavily (24+), especially when the context window is above the limit, on a single machine (with custom llm gateway): it fails 4x more often, and is 2x slower than gpt-5.6.
Both DS and GLM had the same numbers of subagents, 5 or so.
But, well, number of subagents doesn't make a difference if model is dumb (GPT 5.4 High, in May,in Chat mode outperformed what I see with DS4.1-F).
That being said, pricing model makes a huge difference for "find at least one" tasks: with API/PAYG if you have a chance to save 90%, you go for it, whereas with subscriptions it is optimal to burn all your remaining allowance right before reset
Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
How does GLM 5.3 Flash rank against it's big brother and the latest Deepseek?
Both these models are not bad but you can not trust them to make good decisions. I meant GLM 5.1/5.2/5.3 in the previous answer, not the flash version
Worth saying that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (https://z.ai/blog/glm-5.3-flash).
I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.
What I find surprising is that DS Flash can do it at all.
I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.
Thing is that GLM 5.3 is many multiples the cost to run, and slower.
I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.
So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.
A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.
GLM 5.3 doesn't seem to refuse vuln research (which it classifies as "audit") and is good at it.
Therefore you use for offensive cybersecurity tasks because Daybreak Red/Mythos is pure unobtainium for us mere plebians.
DB Blue thankfully exists, but I suspect you risk a ban if you use it with codebases you neither own nor use
Tl;dr because it's the only model at the level of 5.4~5.6 that doesn't refuse tasks nor risk your oai account getting banned
For plain RE tasks Sol or Astra should work just fine (I think)
GLM-5.3; well, the flash variant at least; you just ask what to do and it complies with no objections :^]
You can run deepseek-v4.1-flash on a local server with 4xH200@141GB for around 180K EUR. It can serve more than one user so it can be a good investment for a company, especially if you need your data to stay local (although I would advise do go a step up and populate with 8xH200 to serve many more users, ~300K EUR). You can run whatever test you want and you won't be limited by a front-end that pre-filters your requests.
I use DS straight off of DeepSeek's API.
It was very helpful in the aftermath of a client whose WordPress got stuffed up and analysing when it happened, how it happened, and what the entry point was. The American frontier models refused to help because, well, they just refuse to help with that kind of thing.
I've been testing these models with fireworks.ai
They provide serverless access, and if you use fireconnect it's also easy to hook the models up to e.g. codex which allows for fairly clean comparisons
I think they provide the bare model (no filter) but I might be wrong, haven't tried using it for security research yet
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
Inference providers can credibly promise to not train on your data if they are in a position to get sued.
you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.
Which harness were you using?
When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.
Win what? The race to the bottom always has this competitive language.
“If we ban CFCs now the Chinese will win!”
“If we ban chemical weapons, nuclear weapons, etc etc our enemies will triumph! They won’t stop!”
“If we switch to biodegradeable plastic then our rivals will have an advantage.”
“If we dont externalize the costs to our population, then they will, and then will win!”
I think workflows can do the job agents do, 20x cheaper and more predictably and safely. They can completely displace agents, just as HFCs displaced CFCs and then we were able to ban CFCs and phase them out through international COOPERATION. The language of COOPERATION is what saves us vs COMPETITION is all about cutting corners and externalizing costs. Google the Montreal Protocol, Geneva Conventions, Nuclear Non Proliferation Treaty, Unleaded Gasoline etc etc.
Agents have got to be marginalized. They are just popular because the labs need to make a ton of money for their investors and recoup their massive spending on training models.
Those comparisons are pretty irrelevant
You can't compare banning football to banning genetic experiments and say "they are both bans and therefore directly comparable"
No one was talking about football.
We're talking about replacing most uses of Agents with uses of Workflows. That's the key first step, without which the industry will cry "don't regulate us... do you want China to WIN???"
Come on. Lecturing about hubris when your message is damn the consequences, full speed ahead?
If it's a race to build the torment nexus, or a race with a nonzero chance of building the torment nexus by accident, I don't care about winning.
They said that... where?
By dismissing concerns and focusing exclusively competition and "winning the race."
There's a race? I didn't see that at all in the OP
Seems like a race to me.
Seems pretty bold to claim deepseek is the "best hacking model" while providing zero comparisons to other models...
Notice the qualifier "our", that is the one they have access to.
What if there would be a separate category for distilled models?
DeepSeek is underrated. Basically all Chinese models are good enough for day to day coding at this point.
The 2 trillion dollar ROI on anthropic alone?
Good luck with that.
Good luck building something of value with "day to day coding". If you're going to compete in the future economy, you're going to compete at the frontier. The low hanging fruit will be of no value. If I can recreate your SaaS with an open weight model at a fraction of the cost, then your SaaS has no value.
If I can recreate your "frontier" SaaS with day-to-day coding, it has no value. Or perhaps the few months' lead was never the deciding factor.
the vast majority of successful SaaS offer capabilities or services no where near the frontier.
furthermore i'd argue that the success of an saas, or really any startup, has more to do with things aside from the code and service itself.
Finance management, social connections and effects, word of mouth, networking -- probably all more important than whether or not the todo app uses a functional language and is formally proven.
What the hell are you yammering about. Everything in the economy exists at different quality/price tradeoffs, everything from tomato sauce at the grocery store to landscaping services to SaaS. "Compete at the frontier" is some maximalist buzzword nonsense. Do you think Ikea is crying because their furniture is cheap and they're not "competing at the frontier"?
As opposed to what? Is enclave.ai signed up for GPT Cyber or Glasswing?
DeepSeek models have such good benchmark performance, amazing pricing, and the team over there seems to be widely considered impressive.
I just haven't found them to be very good? I've had a ton more success with the GLM models (since 5.2 anyway). Maybe I'm just holding it wrong, DS models seem to get stuck in loops or tell me nonsense. GLM feels like budget Claude.
It’s funny, in oh-my-pi I’ve had much better luck with DS 4.1 Flash than GLM 5.3 flash. Glm seems to just constantly reason in circles before attempting a tool call
Depends on a lot of factors, AFAIK Claude Code (TUI Harness) does not work well with Deepseek, but Opencode is surprisingly good.
I have never had looping issues with DeepSeek models. Which provider/serving framework and harness are you using?
My experience is complete opposite from yours. The V4 Flash was already quite good, but V4.1 is really very close to SOTA in my books. I've eval'd these models for weeks against Gemini 3.8, Kimi K3 and Opus 5, and DeepSeek absolutely wins these evals.
It's really crazy. We price per token our customers. If Kimi was about 30% of the price of Opus 5 for the same quality, DeepSeek is 1/10th of a price of Kimi K3. We've come down in price so much that I seriously cannot recommend other models before they reduce their pricing.
And what is really interesting is its programming ability. As I've said in my previous comments, I use agents a lot in my work. Since early Opus days until now I have 7-8 agents working in parallel for different tasks. Rust, design, GEPA, evals, analysis. For a long time Kimi K3 was the best model for this work, and before that GPT 5.5. But I still can't really believe how well DeepSeek works here. I really try to find faults from it, trying to see that it must be doing sloppy work and be worse than the others. But it does not. It finishes every task I give to it. And the cost per task is under a dollar, usually 15-30 cents.
In comparison the same task with Kimi would be 3-15 dollars; sometimes closing to 100. And before that with GPT 5.5 a 800 dollar task was not uncommon if I spent days evaluating models.
Now it's less than a dollar.
For me if the other providers will not drop their prices dramatically in the coming weeks I see no reason to use them. Even with a 200 dollar subscription, paying per token for DeepSeek is better value.
My harness: https://omp.sh/
Have you tried Flash V4.1?
What harness are you using? I've had incredible luck with OMP and Deepseek v4 Flash.
Interesting result, but the writing is very poor.
We ran v4.1 Flash through our evaluations and found it to be smarter and faster than V4 Flash, with a commensurate price bump. Some notes:
- Because it's a heavy reasoner, it sits near Gemini 3.7 Flash on the Pareto front (not as cheap as the price suggests in practice).
- Closer than expected to the top open weights models (GLM 5.3 and Kimi K3) in agentic coding, at lower cost.
- Chinese models have always been strong iterators in an agentic harness. This model is no different, reaching an average percentile ~20% higher when given a harness vs a one-shot solution. That one-shot fluid intelligence is what makes a model feel smart, though, and typically results in fewer attempts/tokens to solve a problem, and American frontier models are still far ahead in that department.
The new architecture is interesting. It puts pricing between their old Flash and Pro lineups, suggesting they might be abandoning their super-cheap flash models (which weren't that fast due to heavy reasoning) and their pro models (which sort of flopped and weren't consistently better than their flash models, despite the size/cost) and shipping a strong intermediate that competes with the Gemini Flash series.
Data at https://gertlabs.com/rankings
How do I make deepseek "hack" my source code? do I just start my coding agent in my directory and command it to "find vulnerabilities", or is there some more sophisticated software to do that?