Inversion Triplet - A Contrastive Backdoor Mitigation Method for Self-Supervised Vision Encoders
摘要
Traditional mitigation approaches for backdoored neural networks, such as unlearning and finetuning, face significant challenges when applied to self-supervised learning (SSL) models. Unlearning through neuron pruning often causes catastrophic loss of legitimate features, while finetuning fails to effectively update backdoor neurons due to their dormant nature during clean sample processing. Both approaches distort the learned feature distributions due to the lack of label supervision in SSL settings. We propose Inversion Triplet, a novel mitigation method that leverages contrastive learning principles to overcome these limitations. Our key insight is that by treating clean samples as positive pairs and backdoored samples as negative pairs in a triplet loss formulation, we can isolate and eliminate poison feature clusters while preserving the semantic structure of clean representations. This approach requires neither white-box model access nor label supervision, making it practical for real-world deployment. Extensive experiments demonstrate that Inversion Triplet reduces attack success rates compared to existing methods while maintaining best model utility on clean data.