Hi all, CTO of TypeDB here. The question I actually wanted to answer was "does a type system help agents with correctness?" Everyone's intuition says yes, especially as complexity grows, but evidence is thin and contested, and I haven't found much that isolates the effect for LLMs beyond constrained-decoding work.
A database is a smaller surface than a programming language, so we turned it into a benchmark that holds the model constant and swaps the language: SQL, Cypher, TypeQL, same questions, same data. It's not a clean apples-to-apples comparison and the post's limitations section outlines that further.
One result that points us in the direction of a strong type system is that SQL wins first-shot, but 93% of TypeQL's wrong queries fail with an error, versus 5% for SQL and 40% for Cypher. Add in a retry loop TypeQL ends up ahead, because it can only fix mistakes it can see.
Of course there's a lot of variables here, such model, skill, token usage, question selection, etc. I actually think it's quite a hard thing to study!
If anyone knows of work on LLMs driving systems with stronger vs weaker verifiers, with the model held fixed, I'd love to see it.
Hi all, CTO of TypeDB here. The question I actually wanted to answer was "does a type system help agents with correctness?" Everyone's intuition says yes, especially as complexity grows, but evidence is thin and contested, and I haven't found much that isolates the effect for LLMs beyond constrained-decoding work.
A database is a smaller surface than a programming language, so we turned it into a benchmark that holds the model constant and swaps the language: SQL, Cypher, TypeQL, same questions, same data. It's not a clean apples-to-apples comparison and the post's limitations section outlines that further.
One result that points us in the direction of a strong type system is that SQL wins first-shot, but 93% of TypeQL's wrong queries fail with an error, versus 5% for SQL and 40% for Cypher. Add in a retry loop TypeQL ends up ahead, because it can only fix mistakes it can see.
Of course there's a lot of variables here, such model, skill, token usage, question selection, etc. I actually think it's quite a hard thing to study!
If anyone knows of work on LLMs driving systems with stronger vs weaker verifiers, with the model held fixed, I'd love to see it.