Audio Visual Based Action Recognition
摘要
Action recognition in videos plays an important role in areas like surveillance, sports analytics, and other intelligent systems. Models that rely only on visual input often run into problems, especially when parts of the scene are blocked or when two actions look very similar. In this work, we look at how combining audio and visual cues may help with these limitations. Three of the implementations are based on these four classes from the UCF101 dataset—BoxingPunchingBag, Archery, PlayingCello, and PlayingDaf to keep the setup manageable. While one implementation uses a different set: PlayingSitar, PlayingFlute, SkyDiving, Typing, and TableTennisShot. Visual features are taken from pretrained backbones such as ResNet50, ViT, EfficientNetB0, and MobileNetV3Small. The audio features come from log-mel spectrograms and Wav2Vec2 embeddings. We tested two fusion approaches: simple concatenation and Gated Fusion across several classifiers, including a Temporal Convolutional Network (TCN), a Multilayer Perceptron (MLP), a Vision Transformer with audio, and a Bidirectional LSTM (Bi-LSTM). Among all the models, the Bi-LSTM with Gated Fusion achieves the highest accuracy at 99%. The ViT-based multimodal model follows at 98%, and the MLP and TCN reach 70.08% and 57%. These results suggest that incorporating temporal information and using an adaptive fusion method appears to make a noticeable difference in action recognition. Even though the experiments use a small subset of classes, the findings still point to the potential of audio-visual integration for real-world, scalable applications. Future work aims to extend this approach to the full UCF101 dataset and move toward real-time deployment.