The use of deep learning and anomaly detection methods is of great importance for real-time surveillance, which requires fast detection of suspicious events from streams of multimodal data. Therefore, the models trained for this purpose must be lightweight and computationally efficient. This research explores a multi-modal fusion approach for detecting anomalies using the XD-Violence dataset, integrating video and audio streams to improve detection accuracy. We have proposed the Self-Attention Fusion framework for anomaly detection. This architecture uses pre-trained Resnet-50 weights, BiGRU, and GRU to create video and audio feature vectors, which are passed through a multi-head self-attention layer to give a final output. Our results demonstrate that the Self-Attention Fusion Framework outperforms many existing techniques on the same dataset, achieving a higher AP value of 93.08%, thus validating the effectiveness of attention-based feature integration.