The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages

  • Ángela González de Diego,
  • Franz Wotawa

摘要

The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.