My confirmation bias is satisfied yet again...

Recently I've shared my experience with AI coding assistants shining in smart refactoring while they fail to impress in more complicated stuff requiring a deep understanding of the code base (SourceGraph's Cody AI, Google's Project IDX).

In this recent study of generative AI tools in software development by McKinsey, conducted on a pool of 40 developers, there were little-to-no gains on complex tasks. On the contrary simpler tasks (aka "code generation, refactoring, documentation") saw most of the time savings when developers used AI coding assistants:

Another study from 2022 by the GitHub Copilot team evaluated how the task of creating a simple JavaScript Web server, which typically took ~3 hours (161 minutes to be precise), got accelerated. In a control group of developers who used Copilot time got cut to ~1 hour (71 minutes).

So far I have seen no evidence of successful employment of Coding AI assistants in big/complex coding tasks. Neither in research available in open sources nor in my own experience in a 60 000 software engineering company.

Assistants, such as GitHub's CoPilot X and SourceGraph's Cody AI, are moving in the direction of taking on larger scopes of tasks. Their tech has already stepped beyond GPT-based text autocompletion (as it was in the original CoPilot and Amazon's CodeWhisperer) and chat conversations. Let's see if they manage to bring understanding of the solution to AI and have it reliably change and verify multiple parts of the code base.

Yet developers can start benefiting from AI already. AI coding skills can give small gains and edge to individuals. Pick a tool for your favourite IDE and start typing and prompting! Don't expect AI to magically read your mind and provide large snippets of bulletproof code and you'll be good :)

P.S.: I've seen improvements in Cody AI, they've recently introduced embeddings in their code base understanding tech. Worse a deeper trial and a separate 'Part 2' post.