Scholarlink

Chest X-ray classification is limited by scarce annotations and the heavy cost of transformer models. We propose a three-stage framework: (1) self-supervised pretraining of a Vision Transformer (ViT) with DINOv2 on 880k unlabeled radiographs, (2) fine-tuning on ChestX-ray14, and (3) knowledge transfer into MobileViT using a combination of Binary Cross-Entropy (BCE) and Multi-Label Distillation (MLD) loss. The distilled MobileViT achieves a mean AUROC of 0.8404, surpassing its supervised counterpart by 1.9% while requiring only 0.31 GFLOPs. These results highlight the benefit of domain-specific self-supervised pretraining and demonstrate that BCE and MLD effectively transfers knowledge from large ViTs into compact models. The outcome is a lightweight yet accurate system that can be applied in clinical and resource-constrained healthcare environments.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

When Self-supervised Transformers Meet Knowledge Distillation: Efficient Chest X-Ray Classification

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Quoc-Khang Tran,
  • Nguyen-Khang Pham

摘要

Chest X-ray classification is limited by scarce annotations and the heavy cost of transformer models. We propose a three-stage framework: (1) self-supervised pretraining of a Vision Transformer (ViT) with DINOv2 on 880k unlabeled radiographs, (2) fine-tuning on ChestX-ray14, and (3) knowledge transfer into MobileViT using a combination of Binary Cross-Entropy (BCE) and Multi-Label Distillation (MLD) loss. The distilled MobileViT achieves a mean AUROC of 0.8404, surpassing its supervised counterpart by 1.9% while requiring only 0.31 GFLOPs. These results highlight the benefit of domain-specific self-supervised pretraining and demonstrate that BCE and MLD effectively transfers knowledge from large ViTs into compact models. The outcome is a lightweight yet accurate system that can be applied in clinical and resource-constrained healthcare environments.