Since its inception, ChatGPT has emerged as a potent tool in various Natural Language Processing (NLP) tasks in several domains including health care, education, Business, management, etc. In particular, the education sector is experiencing significant utilization of ChatGPT. The automated correction of code plays a crucial role in the educational process within eLearning platforms and is highly valued by programmers engaged in programming activities. In this study, we conducted an analysis of the performance of ChatGPT in the automated correction and evaluation of small Python code snippets that contain syntax errors. The performance of ChatGPT in correcting and evaluating single line code has been observed to be exceptional, with minimal errors. Nevertheless, as the number of lines of code and errors increases, the performance experiences a decline. However, it is noteworthy that the system still achieves a performance rate exceeding 90% across all potential scenarios. It has been observed that ChatGPT demonstrates enhanced response quality through reinforcement learning when provided with multiple prompts, in contrast to its initial performance at the beginning of prompt execution. Furthermore, it has been observed that while ChatGPT is capable of providing corrections for specific code segments, it is not always reliable in accurately evaluating that corrected code to produce the appropriate result.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance Analysis of ChatGPT in Erroneous Python Code Correction and Evaluation

  • Isha Ganguli,
  • Satyam Atreja,
  • Sanya Bhasin

摘要

Since its inception, ChatGPT has emerged as a potent tool in various Natural Language Processing (NLP) tasks in several domains including health care, education, Business, management, etc. In particular, the education sector is experiencing significant utilization of ChatGPT. The automated correction of code plays a crucial role in the educational process within eLearning platforms and is highly valued by programmers engaged in programming activities. In this study, we conducted an analysis of the performance of ChatGPT in the automated correction and evaluation of small Python code snippets that contain syntax errors. The performance of ChatGPT in correcting and evaluating single line code has been observed to be exceptional, with minimal errors. Nevertheless, as the number of lines of code and errors increases, the performance experiences a decline. However, it is noteworthy that the system still achieves a performance rate exceeding 90% across all potential scenarios. It has been observed that ChatGPT demonstrates enhanced response quality through reinforcement learning when provided with multiple prompts, in contrast to its initial performance at the beginning of prompt execution. Furthermore, it has been observed that while ChatGPT is capable of providing corrections for specific code segments, it is not always reliable in accurately evaluating that corrected code to produce the appropriate result.