Wow, they are at best cherry picking the article they cite on productivity with AI, if not outright trying to lie. They say
Last year, an METR study found that open source coders using AI tools were 19 percent less productive than those who didn’t use them, even as those AI-using coders said they felt 20 percent more productive. But in a February update on a follow-up study, those same researchers said that “we believe it is likely that developers are more sped up from AI tools now—in early 2026—compared to our estimates from early 2025,” citing early raw results and conversations with study participants.
But if you, y’know, actually read this February update, you would see that the researchers are lamenting the fact that they cannot draw conclusions effectively because of the proliferation of AI use. To wit, they say:
Altogether, these issues make it challenging to interpret our central estimate, and we believe it is likely a bad proxy for the real productivity impact of AI tools on these developers.
In fact here is the full passage Arstechnica cherry-picked from (emphasis mine):
Based on conversations with study participants, we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase.
Based on conversations, which is to say, not on their data. This after the study a year prior which suggested developers are not reliable reporters of their own productivity.
Yeah, METR is an industry group funded by Google, Microsoft, Anthropic etc. I happened to summarize their work just yesterday actually: https://lemmy.blahaj.zone/comment/21358850 but one thing I think it’s important to point out is that there is no second study, not yet. The blog, and the open source dataset, are all we have, because the researchers decided to adjust their method. So the intimation arstechnica gives that they completed a second study is false.
And the selection effects they are talking about are pro-ai software developers being unwilling or unable to complete complex tasks without their precious ai anymore, whereas a year ago they were more willing to do both ways in order to prove how powerful ai is. In my opinion, as I stated, this reflects poorly on their levels of self confidence from using ai, more than saying anything about a speed boost/decline.
Wow, they are at best cherry picking the article they cite on productivity with AI, if not outright trying to lie. They say
But if you, y’know, actually read this February update, you would see that the researchers are lamenting the fact that they cannot draw conclusions effectively because of the proliferation of AI use. To wit, they say:
In fact here is the full passage Arstechnica cherry-picked from (emphasis mine):
Based on conversations, which is to say, not on their data. This after the study a year prior which suggested developers are not reliable reporters of their own productivity.
Yeah, METR is an industry group funded by Google, Microsoft, Anthropic etc. I happened to summarize their work just yesterday actually: https://lemmy.blahaj.zone/comment/21358850 but one thing I think it’s important to point out is that there is no second study, not yet. The blog, and the open source dataset, are all we have, because the researchers decided to adjust their method. So the intimation arstechnica gives that they completed a second study is false.
And the selection effects they are talking about are pro-ai software developers being unwilling or unable to complete complex tasks without their precious ai anymore, whereas a year ago they were more willing to do both ways in order to prove how powerful ai is. In my opinion, as I stated, this reflects poorly on their levels of self confidence from using ai, more than saying anything about a speed boost/decline.