Facial Expression Recognition (FER) is becoming one of the most widely utilized techniques for defining the emotional state of a vehicle operator to prevent traffic accidents. Deep CNN networks are heavily utilized for FER tasks, as they have achieved significant advancements and proved their efficiency during the last decade. However, deep CNN networks are computationally expensive due to a significant number of parameters and do not achieve high accuracy in facial expression classification. To address these issues, we optimized the widely-known ResNet-50 and ResNet-101 models by implementing a patch extraction block and a self-attention network. As a result, the optimized models achieved the test accuracies of 95% and 94%, respectively. Additionally, the number of model parameters reduced by 6.17 and 5.87 times, respectively, without any impact on accuracy.