- 217comments
- 628comments
- 18comments
- 4comments
- 1comments
- 221comments
- 48comments
- 2comments
- 9comments
- 64comments
- 625comments
- 65comments
- —discuss
- 37comments
- 24comments
- 5comments
- 152comments
- 2comments
- 74comments
- 143comments
- 2comments
- 368comments
- 305comments
- 225comments
- 100comments
- 30comments
- 181comments
- 28comments
- 27comments
- 65comments
+ Playable Agent driven Starcraft
Dibs on the fly brain
A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?
I wonder if there is a library to decipher brood war replay files. Perhaps an agent could learn by watching.
I haven’t checked for SC:BW but Blizzard has official parsers/replayers for SC2 on GitHub.
Not hard to build. I was shocked at how fast/easy this was to pull together.
This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.
Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.
Oh sorry I should be more clear on that. Will add to report.
For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.
For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.
In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.
Unrelated to the benchmark...
I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...
I miss those days so much.
Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
Later we even listened to Eminem and played violent video games. Most of have never even been charged with a crime, much less abused someone.
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on
[1] https://rolandgao.com/blog/gobench/
[2] https://rolandgao.com/gobench.pdf
Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.
I might open this up to a tournament if enough people want. Any interest?
Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs
I predict LLMs will reach superhuman level and beat even that model in the next 12 months
Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?
It sounds like it might have been actually played in real time, which would be very important to distinguish.
I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE
There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.