XMTF: Cross-Modal Fusion with DINOv2 for Cyberbullying Detection in Code-Mixed Memes
摘要
Cyberbullying has become a pervasive threat on social media platforms, often conveyed through complex, multimodal content such as memes that blend visual elements with text. Conventional unimodal detection systems, which focus solely on text or image, are inadequate in identifying these nuanced signals. To address this, we introduce a deep learning framework that combines RoBERTa for textual feature extraction and Vision Transformers (ViT) pretrained with DINOv2 for visual representation learning. The proposed model employs RoBERTa for textual feature extraction and Vision Transformers (ViT) with DINOv2 for visual representation learning. These modality-specific features are then fused using a Cross-Modal Transformer Fusion (XMTF) module that enables fine-grained semantic alignment between text and image components. This architecture is trained and evaluated on the MultiBully dataset, which consists of memes annotated for cyberbullying in code-mixed Hindi-English, a challenging yet realistic representation of online abuse. The experimental results clearly indicate that our method out-performs both unimodal models and early fusion strategies, achieving an accuracy of 89%, a precision of 87%, a recall of 84%, an F1-score of 85%, and an AUC-ROC of 91%. These outcomes highlight the strength of combining multimodal features through transformer-based fusion for effective cyberbullying detection in practical scenarios.