In social media crowdsourced annotation, image-text multimodal data grows rapidly, but user-generated labels often contain noise (e.g., mislabeling a wolf as a husky). This issue is particularly prominent during the cold start phase of new users, where few-shot data with noisy labels exerts a more significant influence due to the limited data for robust learning. To address this, we propose a multimodal approach for correcting noisy labels. First, we leverage the image-text matching capability of the pre-trained CLIP model to filter mismatched multimodal data and correct erroneous labels via similarity metrics. However, since CLIP’s correction results are not entirely accurate, we introduce a multi-label mechanism that incorporates the top-k labels filtered by CLIP with weighted integration into training, forming a plug-and-play multimodal noisy label correction method. This provides a solution to the cold start problem with noisy behaviors. We embedded our method in 11 datasets and two classic methods, and experimental results validate its effectiveness.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

NLCC: Noisy Label Correction with CLIP for Robust Few-Shot Learning

  • Bowen Han,
  • Shizhuo Deng,
  • Jiaqi Chen,
  • Zehua Gan,
  • Dongyue Chen,
  • Yue Zhu

摘要

In social media crowdsourced annotation, image-text multimodal data grows rapidly, but user-generated labels often contain noise (e.g., mislabeling a wolf as a husky). This issue is particularly prominent during the cold start phase of new users, where few-shot data with noisy labels exerts a more significant influence due to the limited data for robust learning. To address this, we propose a multimodal approach for correcting noisy labels. First, we leverage the image-text matching capability of the pre-trained CLIP model to filter mismatched multimodal data and correct erroneous labels via similarity metrics. However, since CLIP’s correction results are not entirely accurate, we introduce a multi-label mechanism that incorporates the top-k labels filtered by CLIP with weighted integration into training, forming a plug-and-play multimodal noisy label correction method. This provides a solution to the cold start problem with noisy behaviors. We embedded our method in 11 datasets and two classic methods, and experimental results validate its effectiveness.