When learning a new programming language, students can benefit from feedback on their work, to get a better overview of their level, strengths and shortcomings. However, giving personalised feedback on misconceptions is complex due to a lack of means or resources; the design of automated tests and feedback is cumbersome and time-consuming. Our work aims to overcome some of these limitations by enabling automatic feedback thanks to a machine learning model. We developed a multi-label classification architecture following latest advances in natural language processing. By using code embeddings, i.e. generated vectors on students’ code submissions, our system allows to detect specific misconceptions occurring in code snippets, and provide predefined feedback based on these classes. To control the classes and enable the training of our deep neural network, we developed an approach inspired by DeepBugs. The training instances are mutants of original students’ submissions, where the injected modifications are representative of a set of 14 misconceptions we selected. Our model obtained f1-score values up to \(72.9\%\) when predicting an evaluation dataset of students’ mistakes. We also highlight limits of our current mutation labelling technique and improvements to be conducted as further work.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Feedback with BERT: When Detecting Students’ Misconceptions Becomes Automatic

  • Guillaume Steveny,
  • Julien Lienard,
  • Kim Mens,
  • Siegfried Nijssen

摘要

When learning a new programming language, students can benefit from feedback on their work, to get a better overview of their level, strengths and shortcomings. However, giving personalised feedback on misconceptions is complex due to a lack of means or resources; the design of automated tests and feedback is cumbersome and time-consuming. Our work aims to overcome some of these limitations by enabling automatic feedback thanks to a machine learning model. We developed a multi-label classification architecture following latest advances in natural language processing. By using code embeddings, i.e. generated vectors on students’ code submissions, our system allows to detect specific misconceptions occurring in code snippets, and provide predefined feedback based on these classes. To control the classes and enable the training of our deep neural network, we developed an approach inspired by DeepBugs. The training instances are mutants of original students’ submissions, where the injected modifications are representative of a set of 14 misconceptions we selected. Our model obtained f1-score values up to \(72.9\%\) when predicting an evaluation dataset of students’ mistakes. We also highlight limits of our current mutation labelling technique and improvements to be conducted as further work.