Code-related pre-trained language models have achieved state-of-the-art performance in code summarization, and these models are trained and evaluated on large cross-domain datasets. But in the real-world software development scenarios, people are more concerned with generating project-specific summaries for the projects they are working on. While these models perform well in cross-domain datasets, our experiments reveal that they struggle to adapt to specific projects. Notably, data from the target project has a significant impact on improving performance. Unfortunately, many projects, particularly in their early stages, lack sufficient data for effective fine-tuning. In scenarios where project-specific data is limited, using data from projects within the same domain could be a potential solution. However, this approach has not been explored in prior work. This paper further investigates whether domain-specific data can improve model performance for a target project. Additionally, we also explore the effects of a second pre-training phase using different datasets on model performance. Our results show that training with a small amount of domain-specific data can yield comparable performance to models trained with large cross-project datasets. Moreover, a second pre-training phase further enhances the model’s performance on target projects.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Low-Resource Project-Specific Code Summarizaiton: On the Importance of Datasets

  • Rui Peng,
  • Xiangping Chen,
  • Yuan Huang,
  • Xiaocong Zhou

摘要

Code-related pre-trained language models have achieved state-of-the-art performance in code summarization, and these models are trained and evaluated on large cross-domain datasets. But in the real-world software development scenarios, people are more concerned with generating project-specific summaries for the projects they are working on. While these models perform well in cross-domain datasets, our experiments reveal that they struggle to adapt to specific projects. Notably, data from the target project has a significant impact on improving performance. Unfortunately, many projects, particularly in their early stages, lack sufficient data for effective fine-tuning. In scenarios where project-specific data is limited, using data from projects within the same domain could be a potential solution. However, this approach has not been explored in prior work. This paper further investigates whether domain-specific data can improve model performance for a target project. Additionally, we also explore the effects of a second pre-training phase using different datasets on model performance. Our results show that training with a small amount of domain-specific data can yield comparable performance to models trained with large cross-project datasets. Moreover, a second pre-training phase further enhances the model’s performance on target projects.