METR Adjusts Developer Productivity Experiment Design Due to Unreliable AI Impact Data
By
ej88
An everything bagel for the brain. Substantive, layered, well-seasoned.
Summary
METR is changing its developer productivity experiment design after finding that new data from August 2025 gives an unreliable signal about AI tools' impact on productivity. This follows their previous paper which found AI tools caused a 20% slowdown in task completion among experienced open-source developers from February to June 2025. The organization is adjusting its experimental approach due to participant feedback and survey results indicating data reliability issues with the current study design.
Key quotes
· 3 pulledMETR previously published a paper which found the use of AI tools caused a 20% slowdown in completing tasks among experienced open-source developers
To understand how AI is impacting developer productivity over time, we started a new experiment in August 2025 with a larger pool of developers using the latest AI tools
Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools
You might also wanna read
Study finds most developers refuse to code without AI, raising quality concerns
A February 2026 study by AI research lab METR reveals that most developers now refuse to work without AI coding tools. While these tools hel
techcrunch.com·1d ago
MIT study finds 47% drop in brain activity when using AI writing tools, raising concerns about cognitive delegation
An article examining the cognitive costs of AI-assisted writing, citing an MIT Media Lab study showing a 47% drop in brain activity (measure
uxdesign.cc·4d ago
Tokenmaxxing fails as AI ROI metric; workflow redesign needed for productivity gains
The article argues that 'tokenmaxxing'—using token usage as a proxy for employee AI innovation—has failed to deliver meaningful ROI for comp
fortune.com·3d ago