HinglishEval: Evaluating the Effectiveness of Code-Generation Models on Hinglish Prompts
摘要
Code-generation Models are Large Language Models (LLMs) that are fine-tuned to generate code from natural-language prompts. Prior work shows that such models can democratize programming by translating clear English-language prompts into code for simple programming tasks. In this paper, we explore whether this democratization can extend to novice programmers who lack proficiency in English but are able to craft clear prompts in another language. Specifically, we consider prompts in Hinglish, a mixture of Hindi and English that many students in India are comfortable with. We make two contributions. First, we propose a semi-automated technique to translate English prompts into Hinglish, and we use this technique to create HinglishEval: a Hinglish translation of the widely-used code-generation benchmark HumanEval. Second, we compare the performance of several popular open- and closed-source code-generation models on Hinglish and English prompts. Our findings suggest that although code-generation models are generally more effective at generating accurate code from English prompts, their efficacy with Hinglish prompts is promising.