Gartner "Magic Quadrant for AI Code Assistants" (2024) https://www.gartner.com/doc/reprints?id=1-2IKO4MPE&ct=240819...

Additional criteria for assessing AI code assistants from https://news.ycombinator.com/item?id=40478539 re: Text-to-SQL bemchmarks :

codefuse-ai/Awesome-Code-LLM > Analysis of AI-Generated Code, Benchmarks: https://github.com/codefuse-ai/Awesome-Code-LLM :

8.2. Benchmarks: * Integrated Benchmarks, Program Synthesis, Visually Grounded Program Synthesis, Code Reasoning and QA, Text-to-SQL, Code Translation, Program Repair, Code Summarization, Defect/Vulnerability Detection, Code Retrieval, Type Inference, Commit Message Generation, Repo-Level Coding*

OT did not assess:

Aider: https://github.com/paul-gauthier/aider :

Aider works best with GPT-4o & Claude 3.5 Sonnet and can connect to almost any LLM.

https://aider.chat/ :

Aider has one of the top scores on SWE Bench. SWE Bench is a challenging software engineering benchmark where aider solved real GitHub issues from popular open source projects like django, scikitlearn, matplotlib, etc

SWE Bench benchmark: https://www.swebench.com/