Coding is becoming more crucial across numerous scientific fields. The ability to code is increasingly vital for scientists because it enables them to analyze and explore their data in new ways and develop innovative solutions, often much faster than traditional methods. However, many scientists lack competent coding skills. Large Language Models (LLMs) may help bridge this skill gap by helping scientists generate functional, performant code. Our goal was to explore whether or not a scientist who is not familiar with programming can reasonably use an LLM to generate their simulation code. We asked five LLM chatbots to write code to solve a numerical benchmark problem with which we have previously worked, and we measured the performance of the output. We did this across five languages with four versions of the code for both sequential and parallel implementations. We found that the current LLMs are still lacking. Only 47 of the 100 versions created actually compiled and ran with sufficient accuracy for us to proceed with performance testing. Only 31 of the 100 were accurate enough to be considered usable for scientific work. Most of those are significantly slower than the human-written versions. We also used a vibe-coding approach in Cursor to try to optimize the human-written code using parallel kD-trees for each of the five languages. This was more successful with speed boosts seen in three of the five languages.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance of LLM-Generated Code

  • Lisa L. Lacher,
  • Mark C. Lewis,
  • Elizabeth M. Ruetschle,
  • Amanda A. Sickafoose

摘要

Coding is becoming more crucial across numerous scientific fields. The ability to code is increasingly vital for scientists because it enables them to analyze and explore their data in new ways and develop innovative solutions, often much faster than traditional methods. However, many scientists lack competent coding skills. Large Language Models (LLMs) may help bridge this skill gap by helping scientists generate functional, performant code. Our goal was to explore whether or not a scientist who is not familiar with programming can reasonably use an LLM to generate their simulation code. We asked five LLM chatbots to write code to solve a numerical benchmark problem with which we have previously worked, and we measured the performance of the output. We did this across five languages with four versions of the code for both sequential and parallel implementations. We found that the current LLMs are still lacking. Only 47 of the 100 versions created actually compiled and ran with sufficient accuracy for us to proceed with performance testing. Only 31 of the 100 were accurate enough to be considered usable for scientific work. Most of those are significantly slower than the human-written versions. We also used a vibe-coding approach in Cursor to try to optimize the human-written code using parallel kD-trees for each of the five languages. This was more successful with speed boosts seen in three of the five languages.