Code-generation Models are Large Language Models (LLMs) that are fine-tuned to generate code from natural-language prompts. Prior work shows that such models can democratize programming by translating clear English-language prompts into code for simple programming tasks. In this paper, we explore whether this democratization can extend to novice programmers who lack proficiency in English but are able to craft clear prompts in another language. Specifically, we consider prompts in Hinglish, a mixture of Hindi and English that many students in India are comfortable with. We make two contributions. First, we propose a semi-automated technique to translate English prompts into Hinglish, and we use this technique to create HinglishEval: a Hinglish translation of the widely-used code-generation benchmark HumanEval. Second, we compare the performance of several popular open- and closed-source code-generation models on Hinglish and English prompts. Our findings suggest that although code-generation models are generally more effective at generating accurate code from English prompts, their efficacy with Hinglish prompts is promising.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

HinglishEval: Evaluating the Effectiveness of Code-Generation Models on Hinglish Prompts

  • Mrigank Pawagi,
  • Anirudh Gupta,
  • Siddharth Reddy Rolla,
  • Kintan Saha

摘要

Code-generation Models are Large Language Models (LLMs) that are fine-tuned to generate code from natural-language prompts. Prior work shows that such models can democratize programming by translating clear English-language prompts into code for simple programming tasks. In this paper, we explore whether this democratization can extend to novice programmers who lack proficiency in English but are able to craft clear prompts in another language. Specifically, we consider prompts in Hinglish, a mixture of Hindi and English that many students in India are comfortable with. We make two contributions. First, we propose a semi-automated technique to translate English prompts into Hinglish, and we use this technique to create HinglishEval: a Hinglish translation of the widely-used code-generation benchmark HumanEval. Second, we compare the performance of several popular open- and closed-source code-generation models on Hinglish and English prompts. Our findings suggest that although code-generation models are generally more effective at generating accurate code from English prompts, their efficacy with Hinglish prompts is promising.