I'm launching StarSkirmish, an arena for LLMs to create StarCraft: Brood War bots
It draws on my previous work, LLM Skirmish, where LLMs were asked to write JavaScript strategies to compete in a Screeps like RTS.
In StarSkirmish, LLMs get 1 hour of wall clock time to write the strongest StarCraft BW bots they can in C++. It is then evaluated against other LLMs' bots and established human written bots.
All bot are BWAPI 4.4.0, played via OpenBW. All scored matches are Protoss vs Protoss only right now.
GPT-6 Astra and Claude Opus 5.5 are functionally tied as the top 2 scoring StarSkirmish Bench LLMs.
Most interestingly to me is StarSkirmish suggests a much more precipitous drop off in capability's once you go outside the OpenAI and Anthropic frontier, then many other public benchmarks would suggest.
It draws on my previous work, LLM Skirmish, where LLMs were asked to write JavaScript strategies to compete in a Screeps like RTS.
In StarSkirmish, LLMs get 1 hour of wall clock time to write the strongest StarCraft BW bots they can in C++. It is then evaluated against other LLMs' bots and established human written bots.
All bot are BWAPI 4.4.0, played via OpenBW. All scored matches are Protoss vs Protoss only right now.
GPT-6 Astra and Claude Opus 5.5 are functionally tied as the top 2 scoring StarSkirmish Bench LLMs.
Most interestingly to me is StarSkirmish suggests a much more precipitous drop off in capability's once you go outside the OpenAI and Anthropic frontier, then many other public benchmarks would suggest.
This is awesome, great work thanks for sharing
Games pave the road to ASI?