Assessment on the Effectiveness of GitHub Copilot as a Code Assistance Tool: An Empirical Study
摘要
The recent advancements in Machine Learning have led to significant developments in generative AI and large language models. These innovations present an opportunity for creating new features within development tools. However, as assisted programming becomes more prominent across various platforms, it is crucial to assess the implementation of such tools. The question remains: can these advancements genuinely improve the quality and speed of the development process, or are simply an intriguing buzzword? This study aims to evaluate the effectiveness of GitHub Copilot, an AI-powered code assistant tool, in supporting a team of developers working on diverse projects using multiple programming languages and tools. The impact of GitHub Copilot on development productivity, software quality, and developer satisfaction is assessed via an adapted version of the SPACE methodology (Satisfaction, Performance, Activity, Communication, and Efficiency) enriched with a monetization dimension. Over a twelve-week evaluation period, eight diverse users integrate Copilot into their daily development tasks across different programming languages and projects. Users’ perceptions were registered through Weekly meetings. The findings of the study suggest that GitHub Copilot can be a valuable asset to development processes, resulting in enhancements in satisfaction, performance, efficiency, and monetization dimensions. However, areas for improvement include communication features, unit testing, and addressing potential security concerns. This study demonstrates Copilot’s potential as an effective tool for enhancing software development productivity and quality, providing valuable insights for future research and industry adoption.