Design review is essential for ensuring software architecture quality, yet it often relies on subjective developer judgment without standardized processes. While large language models (LLMs) have shown promise in code quality analysis, prior work has largely focused on function-level code smells, with limited exploration of architecture-level review under evolving requirements. This study evaluates the effectiveness of LLMs, enhanced with Retrieval-Augmented Generation (RAG), in generating refactoring suggestions for architectural adaptation. Using three models with varying reasoning capabilities and three prompting strategies, we conduct cross-case comparisons on representative design defect scenarios. Results show that high-capacity models can produce reasonable suggestions even without retrieval support, while RAG significantly improves output quality for weaker models and enhances reasoning depth for mid-tier ones. However, RAG’s effectiveness strongly depends on the semantic relevance of retrieved content. Our findings demonstrate the potential of RAG-enhanced LLMs for architecture-level design review and propose a practical pipeline for integrating LLM reasoning and retrieval. This work paves the way for future applications in CI/CD pipelines and intelligent design-assist tools.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating Architecture Refactoring Suggestions: A Study of LLMs and Retrieval-Augmented Generation Techniques on Design Review

  • Yi-Hui Lai,
  • Yung-Pin Cheng

摘要

Design review is essential for ensuring software architecture quality, yet it often relies on subjective developer judgment without standardized processes. While large language models (LLMs) have shown promise in code quality analysis, prior work has largely focused on function-level code smells, with limited exploration of architecture-level review under evolving requirements. This study evaluates the effectiveness of LLMs, enhanced with Retrieval-Augmented Generation (RAG), in generating refactoring suggestions for architectural adaptation. Using three models with varying reasoning capabilities and three prompting strategies, we conduct cross-case comparisons on representative design defect scenarios. Results show that high-capacity models can produce reasonable suggestions even without retrieval support, while RAG significantly improves output quality for weaker models and enhances reasoning depth for mid-tier ones. However, RAG’s effectiveness strongly depends on the semantic relevance of retrieved content. Our findings demonstrate the potential of RAG-enhanced LLMs for architecture-level design review and propose a practical pipeline for integrating LLM reasoning and retrieval. This work paves the way for future applications in CI/CD pipelines and intelligent design-assist tools.