Objective evaluation of facial nerve function is crucial for the diagnosis and monitoring of facial palsy treatment. In this work, we evaluate the suitability of the Time Series Feature Extraction using Deep Learning (TSFEDL) library for automatic classification of the degree of impairment based on time series of 21 Points of Interest (POIs) on the patient’s face during a standardized set of facial exercises. Four different deep models (HtetMyetLynn, WeiXiaoyan, ZhangJin, and HuangMeiLing) were evaluated on real clinical data. The results showed that the neural network that included long short-term memory (LSTM) layers achieved the best results with a maximum accuracy of 55.016%. The use of GRUs and a spatiotemporal attention mechanism proved to be an effective strategy that significantly reduces the risk of overtraining neural networks.