Scholarlink

Context <p>Code smells, such as God Class or Data Class, are poor design patterns that negatively impact software quality, particularly maintainability. Large Language Models (LLMs) have demonstrated success across various domains by employing fine-tuning or in-context learning (ICL) to perform specific tasks.</p> Objective <p>This study aims to empirically compare the effectiveness and efficiency of fine-tuning and ICL in code smell detection tasks. Effectiveness is measured using the F1 Score, while efficiency is assessed in terms of training time and data usage.</p> Method <p>The experiments were conducted using three datasets: MLCQ, CoRT, and an artificially constructed dataset. For ICL, models GPT4oMini, Gemini-flash, and CodeLLaMA70B, CodeGPT, and GPT-2 were tested with 0, 1, 3, and 5-shot settings. For fine-tuning, CodeBERT, GraphCodeBERT, and UniXcoder, CodeGPT, and GPT-2 were utilized, and hyperparameter tuning was done. The two compact models, CodeGPT and GPT-2, were applied to both techniques, enabling a controlled comparison in which only the learning technique is changed.</p> Results <p>Fine-tuning consistently outperformed ICL, achieving a mean F1 Score improvement of 26%. However, ICL demonstrated greater efficiency, requiring fewer training samples (a mean difference of 1041.25 samples) and less training time (a mean difference of 442.57 seconds). Furthermore, ICL exhibited better generalization capabilities and more flexible adaptation to changes.</p> Conclusions <p>Fine-tuning is recommended when labeled data is abundant, and high performance is critical, especially for domain-specific datasets. In contrast, ICL is preferable in scenarios with insufficient labeled data, such as for low-resource languages or when a more general-purpose model is needed. Additionally, ICL is advantageous when computational resources and time are limited.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Smelly-shot is all you need: an empirical study to compare in-context learning verses fine tuning for code smell detection

  • 期刊论文

  • 发表日期：

    2026年7月8日

查看全文
  • Nawaf Alomari,
  • Moussa Redah,
  • Ahmad Ashraf,
  • Mohammad Alshayeb

摘要

Context

Code smells, such as God Class or Data Class, are poor design patterns that negatively impact software quality, particularly maintainability. Large Language Models (LLMs) have demonstrated success across various domains by employing fine-tuning or in-context learning (ICL) to perform specific tasks.

Objective

This study aims to empirically compare the effectiveness and efficiency of fine-tuning and ICL in code smell detection tasks. Effectiveness is measured using the F1 Score, while efficiency is assessed in terms of training time and data usage.

Method

The experiments were conducted using three datasets: MLCQ, CoRT, and an artificially constructed dataset. For ICL, models GPT4oMini, Gemini-flash, and CodeLLaMA70B, CodeGPT, and GPT-2 were tested with 0, 1, 3, and 5-shot settings. For fine-tuning, CodeBERT, GraphCodeBERT, and UniXcoder, CodeGPT, and GPT-2 were utilized, and hyperparameter tuning was done. The two compact models, CodeGPT and GPT-2, were applied to both techniques, enabling a controlled comparison in which only the learning technique is changed.

Results

Fine-tuning consistently outperformed ICL, achieving a mean F1 Score improvement of 26%. However, ICL demonstrated greater efficiency, requiring fewer training samples (a mean difference of 1041.25 samples) and less training time (a mean difference of 442.57 seconds). Furthermore, ICL exhibited better generalization capabilities and more flexible adaptation to changes.

Conclusions

Fine-tuning is recommended when labeled data is abundant, and high performance is critical, especially for domain-specific datasets. In contrast, ICL is preferable in scenarios with insufficient labeled data, such as for low-resource languages or when a more general-purpose model is needed. Additionally, ICL is advantageous when computational resources and time are limited.