<p>Configuration settings are crucial for adapting software behavior to meet specific performance objectives. However, the prevalence of misconfigurations, combined with the scale and complexity of available options, makes it difficult to determine which configurations have significant performance implications. In this work, we present CongSense, a lightweight framework that employs Large Language Models (LLMs) to systematically identify performance-sensitive configurations with minimal analysis overhead. CongSense employs multiple LLM agents to emulate the reasoning processes of developers and performance engineers, integrating prompting strategies such as prompt chaining and retrieval-augmented generation (RAG). Across seven open-source Java projects, CongSense achieves an average accuracy of 70.55% in classifying performance-sensitive configurations, surpassing both an LLM-based baseline (58.47%) and the previous state-of-the-art approach (60.87%). Evaluations across three LLMs show consistent improvements in accuracy and precision, demonstrating robustness to the choice of foundation model. Notably, prompt chaining yields up to a 17% improvement in precision while preserving recall. A systematic error analysis of 66 consistently misclassified configurations identifies four failure categories, with over 80% of errors caused by misleading performance-related terminology and ungrounded impact reasoning that lead to false positives. Overall, CongSense substantially reduces the manual effort required for configuration analysis and highlights promising directions for LLM-based software performance research.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Identifying performance-sensitive configurations in software systems with LLM-based agents

  • Zehao Wang,
  • Dong Jae Kim,
  • Tse-Hsun Peter Chen

摘要

Configuration settings are crucial for adapting software behavior to meet specific performance objectives. However, the prevalence of misconfigurations, combined with the scale and complexity of available options, makes it difficult to determine which configurations have significant performance implications. In this work, we present CongSense, a lightweight framework that employs Large Language Models (LLMs) to systematically identify performance-sensitive configurations with minimal analysis overhead. CongSense employs multiple LLM agents to emulate the reasoning processes of developers and performance engineers, integrating prompting strategies such as prompt chaining and retrieval-augmented generation (RAG). Across seven open-source Java projects, CongSense achieves an average accuracy of 70.55% in classifying performance-sensitive configurations, surpassing both an LLM-based baseline (58.47%) and the previous state-of-the-art approach (60.87%). Evaluations across three LLMs show consistent improvements in accuracy and precision, demonstrating robustness to the choice of foundation model. Notably, prompt chaining yields up to a 17% improvement in precision while preserving recall. A systematic error analysis of 66 consistently misclassified configurations identifies four failure categories, with over 80% of errors caused by misleading performance-related terminology and ungrounded impact reasoning that lead to false positives. Overall, CongSense substantially reduces the manual effort required for configuration analysis and highlights promising directions for LLM-based software performance research.