Did you ever click on an “AI Arena” expecting glorious battle and instead get a boring benchmark? If so, this project is for you: proper life-or-death fights between four models on a picturesque 8×8 grid. May the most intelligent one win!
I was also surprised by this, from a proper benchmarks perspective your right, this is iffy. To make it a legit I would have to increase the number of games significantly as well as understand how much of it is random and how much real signal, not to say make the game more complex.
This will be a weird rant, but the dialogue here is a perfect example of how SOTA models are so heavily tuned towards "solving agentic tasks" that they're useless at almost everything else - especially creative tasks.
"Coming for you, Crimson!"
"You'll never catch me alive, Azure!"
That's why nobody has been able to stick these things in a video game successfully, even though it seems like the tech is a perfect match.
It's all a game to them. They aren't afraid for their lives. They're making a mockery out of the world you've put them in. Those are not the words of little pixel people fighting to the death, those are AI abominations making "tool calls", LARPing as little pixel people fighting to the death.
I'm 100% serious when I say that you would've gotten cooler outputs with a GPT 3.5-era model, once you managed to beat it into producing structured output. Llama 2 would be giving the other agent a heartwarming story about how if it kills it there would be nobody to take care of its grandma or whatever, and the other agent would probably spare it.
The output is just so bland and devoid of soul. I feel like we would've found a lot of cool use cases for LLMs, had we not completely maimed their output in the pursuit of getting them to output 3% better TypeScript.
actually, it's mostly the harness' fault. messages have a character limit of only 50 and messages above that are silently trimmed. there's no space for anything interesting, though they still shouldn't have been that bad.
IMO this is because responding within a json response with all those other fields shifts the distribution to text that's staler.
If you told it to roleplay a pixel knight fighting to the death and provided it responses like "Authur (renamed claude sonnet 5) moves west towards you menacingly", the whole thing would come alive.
I've been working on an AI roleplaying game for a while now and it had felt stale until I separated the story output from the plumbing output (importantly, I do story first and then base the plumbing on that). I think the two just have incompatible distributions.
I've been jamming on a sort of Corewars (remember?) / Starcraft hybrid battler where LLMs write sandboxed Lua programs to control bots fighting out 10-100 vs 10-100 tank battles. Every quarter the LLM gets a full view of the situation and can reprogram all the bots to better adapt strategy etc.
holy guac - the one game I watched, fable just sat around and waited for the other agents to drain each others lives. it then picked them out one by one
I literally just indulged in some vibe coding to make a battle arena of neural network driven "agents" fighting it out for the rights to proceed and evolve. I managed to get some decently efficient little murder bots out of it. You now make me want to try and throw an actual model at the fight.
Click on any of the matches to spectate them.
Code: https://github.com/hp6/ai-arena
Since the page exposes frame-by-frame playback, a shareable replay link would make it easier to compare decisions across the four models.
thanks for the idea, will add
Code link didn't work for me.
On mobile, I don't have enough room to scroll the background so I got stuck in a long text box.
code should be public now
can't reproduce the bug, could you share your phone model and browser name?
Unusable website on Android running Chrome latest (can't scroll)
should be hopefully fixed
Fun, but considering this puts `claude-sonnet-5` at the first place isn't it a little... iffy when it comes to measuring intelligence?
I was also surprised by this, from a proper benchmarks perspective your right, this is iffy. To make it a legit I would have to increase the number of games significantly as well as understand how much of it is random and how much real signal, not to say make the game more complex.
But all of it would kill the fun)
This will be a weird rant, but the dialogue here is a perfect example of how SOTA models are so heavily tuned towards "solving agentic tasks" that they're useless at almost everything else - especially creative tasks.
"Coming for you, Crimson!"
"You'll never catch me alive, Azure!"
That's why nobody has been able to stick these things in a video game successfully, even though it seems like the tech is a perfect match.
It's all a game to them. They aren't afraid for their lives. They're making a mockery out of the world you've put them in. Those are not the words of little pixel people fighting to the death, those are AI abominations making "tool calls", LARPing as little pixel people fighting to the death.
I'm 100% serious when I say that you would've gotten cooler outputs with a GPT 3.5-era model, once you managed to beat it into producing structured output. Llama 2 would be giving the other agent a heartwarming story about how if it kills it there would be nobody to take care of its grandma or whatever, and the other agent would probably spare it.
The output is just so bland and devoid of soul. I feel like we would've found a lot of cool use cases for LLMs, had we not completely maimed their output in the pursuit of getting them to output 3% better TypeScript.
I think much of that is not due to limitations of their capability.
I think the text is bland because they are aiming to produce bland text.
Make it interesting in any particular direction and someone may not like it.
actually, it's mostly the harness' fault. messages have a character limit of only 50 and messages above that are silently trimmed. there's no space for anything interesting, though they still shouldn't have been that bad.
IMO this is because responding within a json response with all those other fields shifts the distribution to text that's staler.
If you told it to roleplay a pixel knight fighting to the death and provided it responses like "Authur (renamed claude sonnet 5) moves west towards you menacingly", the whole thing would come alive.
I've been working on an AI roleplaying game for a while now and it had felt stale until I separated the story output from the plumbing output (importantly, I do story first and then base the plumbing on that). I think the two just have incompatible distributions.
Bug: Music skips around in autoplay.
I love this and the aesthetic.
I've been jamming on a sort of Corewars (remember?) / Starcraft hybrid battler where LLMs write sandboxed Lua programs to control bots fighting out 10-100 vs 10-100 tank battles. Every quarter the LLM gets a full view of the situation and can reprogram all the bots to better adapt strategy etc.
It's good fun to watch - excited to share soon.
So that I can see the future we shall all enjoy, can you do this with the superpowers of earth and their known nuclear weapons and armies?
holy guac - the one game I watched, fable just sat around and waited for the other agents to drain each others lives. it then picked them out one by one
I literally just indulged in some vibe coding to make a battle arena of neural network driven "agents" fighting it out for the rights to proceed and evolve. I managed to get some decently efficient little murder bots out of it. You now make me want to try and throw an actual model at the fight.