GitHub Copilot is an incredibly limited tool compared to any real harness, and I'm so tired about all of these studies that claim a lack of efficiency while at the same time doing everything in their power to shoot themselves in the foot.
Also, it's insane to me that this whole article has been written without specifying anywhere the models being used.
Like, of course you're not gonna be productive if all you have is Sonnet 4.6???
Setting aside issues with the sample size and response rate, the current progress within the field shows how difficult it is to study the impact of AI on productivity.
Even if the survey was conducted one year later, I would not find it useful to make inferences about the use of AI tools in September 2026.
GitHub Copilot is an incredibly limited tool compared to any real harness, and I'm so tired about all of these studies that claim a lack of efficiency while at the same time doing everything in their power to shoot themselves in the foot.
Also, it's insane to me that this whole article has been written without specifying anywhere the models being used.
Like, of course you're not gonna be productive if all you have is Sonnet 4.6???
Note the timelines of this study:
4/2024: Baseline metrics
8/2024: Participants given Github Copilot licenses
11/2024: Conducted surveys, 261 people invited and 97 responded in survey
9/2026: Study published
Setting aside issues with the sample size and response rate, the current progress within the field shows how difficult it is to study the impact of AI on productivity.
Even if the survey was conducted one year later, I would not find it useful to make inferences about the use of AI tools in September 2026.