Performance of LLM-Generated Code
摘要
Coding is becoming more crucial across numerous scientific fields. The ability to code is increasingly vital for scientists because it enables them to analyze and explore their data in new ways and develop innovative solutions, often much faster than traditional methods. However, many scientists lack competent coding skills. Large Language Models (LLMs) may help bridge this skill gap by helping scientists generate functional, performant code. Our goal was to explore whether or not a scientist who is not familiar with programming can reasonably use an LLM to generate their simulation code. We asked five LLM chatbots to write code to solve a numerical benchmark problem with which we have previously worked, and we measured the performance of the output. We did this across five languages with four versions of the code for both sequential and parallel implementations. We found that the current LLMs are still lacking. Only 47 of the 100 versions created actually compiled and ran with sufficient accuracy for us to proceed with performance testing. Only 31 of the 100 were accurate enough to be considered usable for scientific work. Most of those are significantly slower than the human-written versions. We also used a vibe-coding approach in Cursor to try to optimize the human-written code using parallel kD-trees for each of the five languages. This was more successful with speed boosts seen in three of the five languages.