Facial expression recognition (FER) plays an important role in human-computer interaction (HCI). Building on the current state of the art, the Dual-Direction Attention Mixed Feature Network (DDAMFN), we propose the Mixed Dual-Direction Attention Mixed Feature Network (MDDAMFN), incorporating a novel Mixed Dual-Direction Attention (MDDA) mechanism to address limitations in the original architecture. This new approach captures a wider range of information, from very local to global, mimicking the human perception of facial expressions. The MDDA mechanism enhances the model’s ability to identify better attention regions, significantly improving inter-class and intra-class predictions. Experimental results on different datasets, which are AffectNet, CAER-S, and FERPlus, show that MDDAMFN not only maintains the lightweight and robust characteristics of its predecessor (DDAMFN) but also achieves superior performance compared to existing models, making MDDAMFN a state-of-the-art model in the field of FER.