Scholarlink

<p>In multi-modal perception tasks, test-phase data often suffers from environmental noise and sensor degradation, which causes distribution shifts from the training phase. Test-time adaptation (TTA) is an emerging unsupervised learning strategy that allows pre-trained models to adapt to new data distributions during testing without requiring source domain data. However, existing TTA methods, primarily designed for single-modal data, often struggle with multi-modal data shifts. They may rely on high-confidence pseudo-labels to update model parameters, leading to worse performance than before fine-tuning when all modalities are corrupted, and can suffer from catastrophic forgetting. To address these issues, we propose a <b>En</b>ergy-<b>g</b>uided <b>Two</b>-stage <b>T</b>est-<b>t</b>ime <b>A</b>daptation (Eng2TTA) framework specifically designed for multi-modal perception. In the first stage, an energy-guided loss function is employed to optimize local model parameters by smoothing class distributions within each batch, thereby reducing overconfidence from noisy pseudo-labels. Concurrently, a memory bank is constructed to store the most representative high-confidence sample features for each class. In the second stage, predictions for low-confidence samples are refined by querying the memory bank using feature similarity, leveraging reliable high-confidence information without requiring additional parameter updates, which effectively mitigates catastrophic forgetting. Our method demonstrates superior robustness in multi-modal tasks, significantly outperforming state-of-the-art methods in scenarios with varying levels of modality corruption, particularly under severe distribution shifts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Energy-guided test-time adaptation for data shifts in multi-modal perception

  • 期刊论文

  • 发表日期：

    2025年5月26日

查看全文
  • Yun Pei,
  • Lingbo Liu,
  • Runqing Jiang,
  • Ye Zhang,
  • Pengpeng Yu,
  • Liang Lin,
  • Yulan Guo

摘要

In multi-modal perception tasks, test-phase data often suffers from environmental noise and sensor degradation, which causes distribution shifts from the training phase. Test-time adaptation (TTA) is an emerging unsupervised learning strategy that allows pre-trained models to adapt to new data distributions during testing without requiring source domain data. However, existing TTA methods, primarily designed for single-modal data, often struggle with multi-modal data shifts. They may rely on high-confidence pseudo-labels to update model parameters, leading to worse performance than before fine-tuning when all modalities are corrupted, and can suffer from catastrophic forgetting. To address these issues, we propose a Energy-guided Two-stage Test-time Adaptation (Eng2TTA) framework specifically designed for multi-modal perception. In the first stage, an energy-guided loss function is employed to optimize local model parameters by smoothing class distributions within each batch, thereby reducing overconfidence from noisy pseudo-labels. Concurrently, a memory bank is constructed to store the most representative high-confidence sample features for each class. In the second stage, predictions for low-confidence samples are refined by querying the memory bank using feature similarity, leveraging reliable high-confidence information without requiring additional parameter updates, which effectively mitigates catastrophic forgetting. Our method demonstrates superior robustness in multi-modal tasks, significantly outperforming state-of-the-art methods in scenarios with varying levels of modality corruption, particularly under severe distribution shifts.