Real-time sports activity recognition is a challenging task in computer vision, especially for fast-paced sports involving complex interactions. This study tackles this issue by presenting a novel deep learning framework specifically designed for real-time activity recognition in volleyball and cricket. The primary aim is to develop and evaluate a high-performance system that outperforms existing methods in both speed and accuracy. Our approach integrates YOLOv8, a leading object detection model, with an Enhanced Feature Pyramid Network (EFPN). This hybrid architecture utilizes YOLOv8’s transformer-based structure for efficient object detection and EFPN’s multi-scale feature maps to improve spatial resolution. Lateral connections and upsampling techniques are employed to enhance feature representation across various scales, essential for accurately capturing diverse sports activities. Extensive evaluations on professional volleyball and cricket datasets reveal significant improvements over baseline methods. The proposed framework achieves a mean Average Precision (mAP) of 99.23% at 23 frames per second for volleyball, and 98.9% mAP at 23 frames per second for cricket. Ablation studies highlight the individual contributions of YOLOv8 and EFPN, offering insights into their combined effectiveness. This framework not only advances real-time sports activity recognition but also has potential applications in automated performance analysis, tactical decision-making, and enhanced sports broadcasting. Future research will focus on challenges like occlusion handling, multi-player interactions, and integration with multimodal data for comprehensive sports analytics.