Facial Expression Recognition (FER) is essential for understanding human emotions, improving human–computer interactions, and supporting mental health monitoring. This work presents a novel FER system that uses the YOLO11 architecture to address key challenges, including computational inefficiency, subtle expression detection, and environmental variations such as occlusion and lighting. FER plays a vital role in applications such as healthcare, education, and intelligent systems, bridging gaps in human–computer interaction and improving emotional intelligence in machines. The proposed system integrates multi-scale detection, feature fusion, and advanced attention mechanisms, such as the C2PSA block, to improve accuracy and robustness. Unlike previous YOLO versions, YOLO11 introduces innovative transformer-based modules and refined attention mechanisms that enhance subtle emotion recognition, particularly under challenging conditions. Using the FER2013 dataset, the system achieved a general mean average precision mAP@50 of 84.0% and mAP@50-95 of 83.9%. High classification accuracy was observed for emotions such as happy and surprise, while lower performance was observed for neutral and sad expressions. These results show that YOLO11 demonstrates improved performance compared to earlier YOLO versions, indicating its suitability for FER in diverse and dynamic conditions. Future work aims to refine subtle emotion detection, expand datasets for greater diversity, and enhance performance for less distinguishable expressions in real-world environments.