Scholarlink

With the extensive employment of large Foundation Models, the traditional classification concepts are challenged. In this work, we target a crucial task in natural language processing (NLP) and understanding (NLU); i.e. intent classification. We first debate the applicability of classical out-of-distribution definition and introduce a more realistic type; i.e. non-semantic OOD (NS-OOD). The results of our experiments asserted our claim and demonstrated that NS-OOD data poses a challenge for even powerful classifiers. Additionally, we support the criticism of relying only on the classifier’s last layer logits to evaluate its correctness. Consequently, we propose a method (ReaL) to regain the trust of softmax score as a confidence indicator. This is performed by distilling the logits using the temperature parameter. We show that ReaL provides better separation between correct and wrong predictions’ scores. To achieve a proper temperature estimation, we perform a mathematical derivation. Our approach calculates an adequate temperature value for a given model and a dataset (Our code is available at https://github.com/modafarshouha/ReaL .).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ReaL: Reassessing Softmax Score by Logits Distillation for Foundation Model Based Intent Classification

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Gábor Szűcs,
  • Modafar Al-Shouha

摘要

With the extensive employment of large Foundation Models, the traditional classification concepts are challenged. In this work, we target a crucial task in natural language processing (NLP) and understanding (NLU); i.e. intent classification. We first debate the applicability of classical out-of-distribution definition and introduce a more realistic type; i.e. non-semantic OOD (NS-OOD). The results of our experiments asserted our claim and demonstrated that NS-OOD data poses a challenge for even powerful classifiers. Additionally, we support the criticism of relying only on the classifier’s last layer logits to evaluate its correctness. Consequently, we propose a method (ReaL) to regain the trust of softmax score as a confidence indicator. This is performed by distilling the logits using the temperature parameter. We show that ReaL provides better separation between correct and wrong predictions’ scores. To achieve a proper temperature estimation, we perform a mathematical derivation. Our approach calculates an adequate temperature value for a given model and a dataset (Our code is available at https://github.com/modafarshouha/ReaL .).