AI Coding Tools Promise Speed, But Slow Pros Down

Artificial intelligence coding assistants are have been touted as the inevitable accelerant for software development for years now, promising to liberate engineers from drudgery and transform the business of making code. But findings from a nonprofit AI research group challenge those assumptions, showing how little is settled about AI’s actual impact on developer productivity.

See Also: Complete Guide to Building an Identity Protection Strategy

A study by nonprofit METR found that AI coding tools slowed experienced developers by 19% despite expectations of faster work. Some teams saw gains, but researchers cautioned against assuming universal productivity boosts, citing reliability and workflow challenges.

In a randomized controlled trial, METR recruited 16 experienced open-source developers to test whether AI tools deliver on their efficiency claims. Participants were assigned 246 tasks across large, mature codebases they regularly maintained. About half the tasks allowed the use of advanced AI coding tools, such as Cursor Pro, while the rest required developers to rely solely on their own expertise.

Before beginning the work, developers forecasted that AI would cut their task completion times by 24%. But the study instead showed the opposite effect. “Surprisingly, we find that allowing AI actually increases completion time by 19%. Developers are slower when using AI tooling,” METR researchers said.

A key factor behind this slowdown appeared to be the reliability of AI-generated code. Human programmers often found themselves spending more time reviewing, testing and ultimately rewriting the code produced by the AI tool, the study said. One participant said he “wasted at least an hour first trying to [solve a specific issue] with AI” before reverting all the changes and completing the work manually. Other studies have shown that AI coding tools can introduce security vulnerabilities and errors into code.

The findings complicate the narrative that tools such as GitHub Copilot, Cursor and similar AI assistants are an unambiguous force multiplier for developers of all skill levels. METR”s study said that AI-driven coding tends to show more promise when building new systems from scratch rather than when extending or maintaining large, complex codebases – the kind of work that occupies much of the professional software world.

The researchers advised caution in generalizing their results, acknowledging both the small sample size and the pace at which AI tools are improving. The nonprofit organization found that AI coding tools have significantly improved their ability to complete complex, long-horizon tasks in recent years and wouldn’t expect the same results to hold true even months from now.

Other studies have reached different conclusions. Large-scale experiments have found measurable productivity benefits for certain teams and use cases. But METR”s trial shows the variability in outcomes depending on developer familiarity, task complexity and tool maturity.

Experience with specific AI tools was another confounding factor. Although nearly all participants had used web-based language models to assist their workflows in the past, only 56% had prior experience with Cursor, the main coding assistant in the trial. All developers received training, but the learning curve may have contributed to the increased time required to complete tasks.

Since the launch of ChatGPT, many industry figures have envisioned a shift to natural language programming, by which engineers simply describe what they want in plain English and watch the code materialize. “The hottest new programming language is English,” OpenAI cofounder Andrej Karpathy had declared in early 2023. He later popularized the idea of “vibe coding,” which is rapid prototyping by instructing an AI tool to generate entire software components.

This vision has fueled speculation that human programmers could soon find themselves relegated to refining specifications and overseeing quality control – or even pushed out of the field altogether. But METR”s results suggest that, at least for now, AI coding tools are unreliable enough that experienced developers must still invest significant effort to validate and correct the output.

The software community is divided over what these trends will ultimately mean. Simon Willison, an AI developer and blogger, recently argued that fears of obsolescence are overblown. “Quitting programming as a career right now because of LLMs would be like quitting carpentry as a career thanks to the invention of the table saw,” he wrote on Bluesky. Industry veteran Tom Coates added that it “could be like quitting agriculture as a career thanks to the invention of tractors and combine harvesters instead.”